{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# `<brilliant seminar title>`\n",
    "\n",
    "In this seminar, we're going to play with [Tensorflow](https://www.tensorflow.org/) and see how it helps us build deep learning models.\n",
    "\n",
    "If you're running this notebook outside course environment, you'll need to install tensorflow:\n",
    "* `pip install tensorflow` should install cpu-only TF on Linux & Mac OS\n",
    "* If you want GPU support from offset, see [TF install page](https://www.tensorflow.org/install/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warming up\n",
    "For starters, let's implement a python function that computes the sum of squares of numbers from 0 to N-1.\n",
    "* Use numpy or python\n",
    "* An array of numbers 0 to N - numpy.arange(N)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np",
    "\n",
    "\n",
    "\n",
    "def sum_squares(N):",
    "\n",
    "    return < student.Implement_me() >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.01 s, sys: 2.94 s, total: 3.96 s\n",
      "Wall time: 9.48 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "662921401752298880"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "% % time",
    "\n",
    "sum_squares(10**8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensoflow teaser\n",
    "\n",
    "Doing the very same thing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf",
    "\n",
    "s = tf.InteractiveSession()",
    "\n",
    "\n",
    "# I gonna be your function parameter",
    "\n",
    "N = tf.placeholder('int64', name=\"input_to_your_function\")",
    "\n",
    "\n",
    "# i am a recipe on how to produce sum of squares of arange of N given N",
    "\n",
    "result = tf.reduce_sum((tf.range(N)**2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "662921401752298880\n",
      "CPU times: user 1.26 s, sys: 564 ms, total: 1.83 s\n",
      "Wall time: 1.79 s\n"
     ]
    }
   ],
   "source": [
    "% % time",
    "\n",
    "# example of computing the same as sum_squares",
    "\n",
    "print(result.eval({N: 10**8}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How does it work?\n",
    "1. define placeholders where you'll send inputs;\n",
    "2. make symbolic graph: a recipe for mathematical transformation of those placeholders;\n",
    "3. compute outputs of your graph with particular values for each placeholder\n",
    "  * output.eval({placeholder:value}) \n",
    "  * s.run(output, {placeholder:value})\n",
    "\n",
    "* So far there are two main entities: \"placeholder\" and \"transformation\"\n",
    "* Both can be numbers, vectors, matrices, tensors, etc.\n",
    "* Both can be int32/64, floats of booleans (uint8) of various size.\n",
    "\n",
    "* You can define new transformations as an arbitrary operation on placeholders and other transformations\n",
    " * tf.reduce_sum(tf.arange(N)\\**2) are 3 sequential transformations of placeholder N\n",
    " * There's a tensorflow symbolic version for every numpy function\n",
    "   * `a+b, a/b, a**b, ...` behave just like in numpy\n",
    "   * np.mean -> tf.reduce_mean\n",
    "   * np.arange -> tf.range\n",
    "   * np.cumsum -> tf.cumsum\n",
    "   * If if you can't find the op you need, see the docs [docs](https://www.tensorflow.org/api_docs/python).\n",
    " \n",
    " \n",
    "Still confused? We gonna fix that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Default placeholder that can be arbitrary float32 scalar, vertor, matrix, etc.",
    "\n",
    "arbitrary_input = tf.placeholder('float32')",
    "\n",
    "\n",
    "# Input vector of arbitrary length",
    "\n",
    "input_vector = tf.placeholder('float32', shape=(None,))",
    "\n",
    "\n",
    "# Input vector that _must_ have 10 elements and integer type",
    "\n",
    "fixed_vector = tf.placeholder('int32', shape=(10,))",
    "\n",
    "\n",
    "# Matrix of arbitrary n_rows and 15 columns (e.g. a minibatch your data table)",
    "\n",
    "input_matrix = tf.placeholder('float32', shape=(None, 15))",
    "\n",
    "\n",
    "# You can generally use None whenever you don't need a specific shape",
    "\n",
    "input1 = tf.placeholder('float64', shape=(None, 100, None))",
    "\n",
    "input2 = tf.placeholder('int32', shape=(None, None, 3, 224, 224))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# elementwise multiplication",
    "\n",
    "double_the_vector = input_vector*2",
    "\n",
    "\n",
    "# elementwise cosine",
    "\n",
    "elementwise_cosine = tf.cos(input_vector)",
    "\n",
    "\n",
    "# difference between squared vector and vector itself",
    "\n",
    "vector_squares = input_vector**2 - input_vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Practice time:",
    "\n",
    "# create two vectors of size float32",
    "\n",
    "my_vector = <student.init_float32_vector() >",
    "\n",
    "my_vector2 = <student.init_one_more_such_vector() >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write a transformation(recipe):",
    "\n",
    "#(vec1)*(vec2) / (sin(vec1) +1)",
    "\n",
    "my_transformation = <student.implementwhatwaswrittenabove() >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'my_transformation' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-33-3cc7edc0e33f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mprint\u001b[0m \u001b[0mmy_transformation\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;31m#it's okay, it's a symbolic graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'my_transformation' is not defined"
     ]
    }
   ],
   "source": [
    "print(my_transformation)",
    "\n",
    "# it's okay, it's a symbolic graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#",
    "\n",
    "dummy = np.arange(5).astype('float32')",
    "\n",
    "\n",
    "my_transformation.eval({my_vector: dummy, my_vector2: dummy[::-1]})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing graphs\n",
    "\n",
    "It's often useful to visualize the computation graph when debugging or optimizing. \n",
    "Interactive visualization is where tensorflow really shines as compared to other frameworks. \n",
    "\n",
    "There's a special instrument for that, called Tensorboard. You can launch it from console:\n",
    "\n",
    "```tensorboard --logdir=/tmp/tboard --port=7007```\n",
    "\n",
    "If you're pathologically afraid of consoles, try this:\n",
    "\n",
    "```os.system(\"tensorboard --logdir=/tmp/tboard --port=7007 &\"```\n",
    "\n",
    "_(but don't tell anyone we taught you that)_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# launch tensorflow the ugly way, uncomment if you need that",
    "\n",
    "#!killall tensorboard",
    "\n",
    "#os.system(\"tensorboard --logdir=/tmp/tboard --port=7007 &\")",
    "\n",
    "\n",
    "# show graph to tensorboard",
    "\n",
    "writer = tf.summary.FileWriter(\"/tmp/tboard\", graph=tf.get_default_graph())",
    "\n",
    "writer.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One basic functionality of tensorboard is drawing graphs. One you've run the cell above, go to `localhost:7007` in your browser and switch to _graphs_ tab in the topbar. \n",
    "\n",
    "Here's what you should see:\n",
    "\n",
    "<img src=\"https://s12.postimg.cc/a374bmffx/tensorboard.png\" width=480>\n",
    "\n",
    "Tensorboard also allows you to draw graphs (e.g. learning curves), record images & audio ~~and play flash games~~. This is useful when monitoring learning progress and catching some training issues.\n",
    "\n",
    "One researcher said:\n",
    "```\n",
    "If you spent last four hours of your worktime watching as your algorithm prints numbers and draws figures, you're probably doing deep learning wrong.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can read more on tensorboard usage [here](https://www.tensorflow.org/get_started/graph_viz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do It Yourself\n",
    "\n",
    "__[2 points max]__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quest #1 - implement a function that computes a mean squared error of two input vectors",
    "\n",
    "# Your function has to take 2 vectors and return a single number",
    "\n",
    "\n",
    "<student.define_inputs_and_transformations() >",
    "\n",
    "\n",
    "mse = <student.define_transformation() >",
    "\n",
    "\n",
    "\n",
    "def compute_mse(vector1, vector2): return < how to run you graph?>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Tests",
    "\n",
    "from sklearn.metrics import mean_squared_error",
    "\n",
    "\n",
    "for n in [1, 5, 10, 10**3]:",
    "\n",
    "\n",
    "    elems = [np.arange(n), np.arange(n, 0, -1), np.zeros(n),",
    "\n",
    "             np.ones(n), np.random.random(n), np.random.randint(100, size=n)]",
    "\n",
    "\n",
    "    for el in elems:",
    "\n",
    "        for el_2 in elems:",
    "\n",
    "            true_mse = np.array(mean_squared_error(el, el_2))",
    "\n",
    "            my_mse = compute_mse(el, el_2)",
    "\n",
    "            if not np.allclose(true_mse, my_mse):",
    "\n",
    "                print 'Wrong result:'",
    "\n",
    "                print 'mse(%s,%s)' % (el, el_2)",
    "\n",
    "                print \"should be: %f, but your function returned %f\" % (true_mse, my_mse)",
    "\n",
    "                raise ValueError, \"Что-то не так\"",
    "\n",
    "\n",
    "print \"All tests passed\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# variables\n",
    "\n",
    "The inputs and transformations have no value outside function call. This isn't too comfortable if you want your model to have parameters (e.g. network weights) that are always present, but can change their value over time.\n",
    "\n",
    "Tensorflow solves this with `tf.Variable` objects.\n",
    "* You can assign variable a value at any time in your graph\n",
    "* Unlike placeholders, there's no need to explicitly pass values to variables when `s.run(...)`-ing\n",
    "* You can use variables the same way you use transformations \n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating shared variable",
    "\n",
    "shared_vector_1 = tf.Variable(initial_value=np.ones(5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "initial value [ 1.  1.  1.  1.  1.]\n"
     ]
    }
   ],
   "source": [
    "# initialize variable(s) with initial values",
    "\n",
    "s.run(tf.global_variables_initializer())",
    "\n",
    "\n",
    "# evaluating shared variable (outside symbolicd graph)",
    "\n",
    "print(\"initial value\", s.run(shared_vector_1))",
    "\n",
    "\n",
    "# within symbolic graph you use them just as any other inout or transformation, not \"get value\" needed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('new value', array([ 0.,  1.,  2.,  3.,  4.]))\n"
     ]
    }
   ],
   "source": [
    "# setting new value",
    "\n",
    "s.run(shared_vector_1.assign(np.arange(5)))",
    "\n",
    "\n",
    "# getting that new value",
    "\n",
    "print(\"new value\", s.run(shared_vector_1))",
    "\n",
    "\n",
    "# Note that the vector changed shape",
    "\n",
    "# This is entirely allowed... unless your graph is hard-wired to work with some fixed shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tf.gradients - why graphs matter\n",
    "* Tensorflow can compute derivatives and gradients automatically using the computation graph\n",
    "* Gradients are computed as a product of elementary derivatives via chain rule:\n",
    "\n",
    "$$ {\\partial f(g(x)) \\over \\partial x} = {\\partial f(g(x)) \\over \\partial g(x)}\\cdot {\\partial g(x) \\over \\partial x} $$\n",
    "\n",
    "It can get you the derivative of any graph as long as it knows how to differentiate elementary operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_scalar = tf.placeholder('float32')",
    "\n",
    "\n",
    "scalar_squared = my_scalar**2",
    "\n",
    "\n",
    "# a derivative of scalar_squared by my_scalar",
    "\n",
    "derivative = tf.gradients(scalar_squared, my_scalar)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8e8d16c510>"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXYAAAD8CAYAAABjAo9vAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4FFXbx/HvSe+BQKgh9F4CIaGDgCCoCIIiRUFUROAR\nCyoqWB5FRMWKWABRRBCpFlQE6U1KAoTeQg01BNLrZs/7x0Re9KGE7CaT3dyf6+KSTXbP3CPwYzhz\n5j5Ka40QQgjn4WJ2AUIIIexLgl0IIZyMBLsQQjgZCXYhhHAyEuxCCOFkJNiFEMLJSLALIYSTkWAX\nQggnI8EuhBBOxs2Mg5YtW1ZXq1bNjEMLIYTDio6Ovqi1Dr7Z+0wJ9mrVqhEVFWXGoYUQwmEppU7k\n530yFSOEEE5Ggl0IIZyMBLsQQjgZCXYhhHAyEuxCCOFkJNiFEMLJSLALIYSTcahgjzp+iS/WxJpd\nhhBC3LLMnFz++8teLqRkFvqxHCrYf9t9lnf/OMDWY5fMLkUIIW7JRysOMXPTcQ6fTy30YzlUsL/Q\nrS5Vgrx5cdEuMnNyzS5HCCHyJeZUItPXHaV/ZBXa1ipb6MdzqGD38XDjnT5NOHYxjY/+PGR2OUII\ncVPZFitjFu6inL8XY++uXyTHdKhgB2hbqywDWoQyff1RYk4lml2OEELc0Gerj3DwfAoTejciwMu9\nSI7pcMEO8PJd9Sgf4MULC2PIssiUjBCieNp/NpnPVh/h3qaVuL1++SI7rkMGe4CXO2/3bsyh86l8\ntlpWyQghih9LrjEFU8rHndfvaVikx3bIYAfoVK8cfZpV5vPVR9h3JtnscoQQ4h+mrz/G7tNJvNGz\nEaV9PYr02A4b7ACv3dOAUj4ejFkUgyXXanY5QggBQGx8Kh+tOET3hhW4q3GFIj++Qwd7KR8P3rq3\nIXtOJzNt/VGzyxFCCHKtmjELd+Ht7sqb9zZEKVXkNTh0sAN0b1SRuxpX4OM/D3P4fIrZ5QghSriZ\nm44TfeIyr/VoQDl/L1NqsEuwK6WeVUrtVUrtUUrNVUoV6dm82asRfl5uPDt/JzkyJSOEMMmRC6m8\n98cBOtcrR5/wyqbVYXOwK6UqA08BEVrrRoAr0N/WcW9FWT9P3u7diD2nk5my6khRHloIIQBjFcxz\n83fi7eHKO30amzIF8zd7TcW4Ad5KKTfABzhjp3HzrXujivRuVpkpq4+wK04eXBJCFK3P18QSE5fE\nW/c2olyAOVMwf7M52LXWp4H3gZPAWSBJa7383+9TSg1TSkUppaLi4+NtPew1/bdnQ4L9PBk9P0Z6\nyQghisye00lMXnmYe8Iq0aNJJbPLsctUTGmgF1AdqAT4KqUe+vf7tNbTtNYRWuuI4OBgWw97TYHe\n7rx3fxOOXEjl/WUHC+UYQghxtcycXEbP30mQrwfjexXtg0jXY4+pmC7AMa11vNY6B1gMtLHDuAXS\noU4wD7UKZcbGY2w+mmBWGUKIEuKjPw9x6Hwq797XhFI+Rfsg0vXYI9hPAq2UUj7KuFtwO7DfDuMW\n2Ni76hMa5MPzC2JIzbKYWYoQwoltO36JaeuPMqBFFTrVK2d2OVfYY459C7AQ2A7szhtzmq3j2sLH\nw40P+oZxOjGDCb/tM7MUIYSTSsuy8Nz8GEJKezPu7gZml/MPdlkVo7V+XWtdT2vdSGs9SGudZY9x\nbRFRLYhhHWowd+spVh04b3Y5QggnM+H3/Zy6nM7794fh5+lmdjn/4PBPnt7I6K51qFfBnxcW7CI+\nxfS/a4QQTmL53nN8v+Ukj7evQcsaZcwu5384dbB7urkyeUAzUrMsvLAwBq212SUJIRzc+eRMXly0\ni4aVAnj+jrpml3NNTh3sAHXK+zPu7vqsORjPzE3HzS5HCOHArFbNc/NjyMjJ5ZP+zfBwK54RWjyr\nsrNBrapye71yTFx6gAPnpHe7EKJgZmw4xoYjF3mtR0NqlfMzu5zrKhHBrpTi3fubEODlzlNzd8hT\nqUKIW7bndBLvLTtAt4blGdCiitnl3FCJCHYwGoV98EAYh86nMvF3U5fZCyEcTEZ2Lk//sIMgXw/e\n6dPE1AZf+VFigh3gtjrBPNq2Ot/+dUKWQAoh8m38b/uIjU/jwweaFvk2dwVRooIdYEz3urIEUgiR\nb38vbRzWoQZta5U1u5x8KXHB7uXuyqd5SyBHz9+J1SpLIIUQ13YmMaPYL228lhIX7AC1y/vz354N\nWX/4Il+sjTW7HCFEMZSTa2XU3B1kW6x8OqD4Lm28Fsep1M76R1ahV9NKfLD8IFukC6QQ4l8+WH6I\n6BOXebtPY2oEF9+ljddSYoNdKcWE3o2pVsaXp37YQUKqzLcLIQyrD1zgy7WxDGgRSq+m5u1dWlAl\nNtgB/DzdmDIwnMvpOTw7P0bm24UQnE3KYPT8ndSr4M/r9xSvro35VaKDHaBBpQBev6cB6w7Fy3y7\nECWcJdfKqO+NefXPHgzHy93V7JIKpMQHO8DAFqHcE2bMt289dsnscoQQJvngz0NE5c2r13SwefWr\nSbBjzLe/3bsRoUE+PDVX5tuFKInWHLzAF2tiGdCiikPOq19Ngj2Pv5c7UwaGcyk9m2fm7SRX5tuF\nKDHOJGYwen5M3rx68diQ2hZ2CXalVCml1EKl1AGl1H6lVGt7jFvUGlUO5L/3GOvbP15xyOxyhBBF\nIDMnlxGzox1+Xv1q9trP6RPgD631/UopD8DHTuMWuQEtqrDj5GU+XXWEJiGl6NqgvNklCSEK0RtL\n9hITl8SXDzV36Hn1q9l8xa6UCgQ6ADMAtNbZWutEW8c1i1KK8fc2onHlQEbP28mxi2lmlySEKCTz\ntp1k7tZTjOxYk+6NKhT+AdOLZnGGPaZiqgPxwDdKqR1Kqa+UUr52GNc0Xu6ufPFQOG6uiuHfRZOe\nbTG7JCGEne2KS+TVn/fSvnZZnivsPjDpl2D5q/BhAzjxV+EeC/sEuxsQDnyhtW4GpAEv/ftNSqlh\nSqkopVRUfHy8HQ5buEJK+zB5QDMOX0jhxUW7Zb9UIZxIQmoWw7+LJtjPk0/6N8PVpZD6q2cmw5p3\n4OMmsOlTqH8PBFQsnGNdxR7BHgfEaa235L1eiBH0/6C1nqa1jtBaRwQHB9vhsIWvfe1gnrujLkti\nzjBjwzGzyxFC2IEl18pTP+zgYlo2Xz7UnKDC6K+enQ4bP4FPwmDNRKhxG4zYBPdNh9LV7H+8f7H5\n5qnW+pxS6pRSqq7W+iBwO7DP9tKKh5EdaxJzKpGJSw/QqHIgrWqUMbskIYQN3l9+iI1HEnjvviY0\nDgm07+CWLNg+C9ZNgtTzUPN26PwKVP6fa91CZa917KOAOUqpXUBT4G07jWs6pRQfPBBG1SAfnvx+\nO6cTM8wuSQhRQL/tOsuXa2MZ2DKUByLtuG9prgW2fwefRsDvz0NQDXhkKQxaXOShDnYKdq31zrxp\nliZa63u11pftMW5x4e/lzrTBzcnKsfL4t1FyM1UIB7TndBLPLdhJeGgp+zX3slphzyL4vCX88iT4\nBMFDi4xQr9rGPscoAHnyNJ9qlfNn8sBmHDiXzHPSCVIIh3IhOZPHZ0UR5OPB1EEReLrZ+BCS1nBw\nKUztAAsfBRd36Dcbhq2BWl3A5M2uJdhvQae65Rh7V32W7jnHxysPm12OECIfMnNyGfZdNInpOUx/\nOIJgf0/bBjy6Br7qAnP7Q3Yq9JkOIzYaK15MDvS/2evJ0xLjsXbVOXguhckrD1OnvB89mlQyuyQh\nxHVorRm7eDc7TyXy5UPhNKxkw83Sk1tg1Xg4vh4CKsM9k6HpQHB1t1/BdiLBfouUUrzVuxHHLqbx\n/IIYqgb52v/OuhDCLqauO8riHacZ3bUO3RsVcP342V2w6i04vAx8g6H7O9D8EXD3sm+xdiRTMQXg\n6ebKl4OaU8bXk8dnRXEhOdPskoQQ/7Ji33ne/eMAPZpUZFTnWrc+QPwhmP8wTG0Pp7bA7a/D0zHQ\nakSxDnWQYC+wsn6eTB8cQXJmDo9/F01mTq7ZJQkh8hw8l8LTP+ygUaVAJt0fhrqVue/Lx+HHEcZK\nlyMroMMYI9DbjwYPx+iWIsFugwaVAvioX1N2xSXy7LydslJGiGLgfHImj3yzFV9PN6YPjsDbI58r\nYJLPwq+jjbXoexZBq5FGoHceB96lCrdoO5Ngt1G3hhUYl7dSZuLS/WaXI0SJlppl4dGZ20jMyOHr\nIZFUCMzHlElaAix/BSY3he3fQvggeHondJsAvmULv+hCIDdP7eCxdtWJu5zB9PXHqFzKmyFtq5td\nkhAljiXXyn/mbOfAuRS+ejiCRpVvsqghMwk2TYHNn0NOOjTpD7eNgSDH//MrwW4HSile7dGA04kZ\nvPHrPiqV8uaOhkXQ21kIARjLGl/9eQ9rD8Xzdu/GdKpb7vpvzk6DLVONJl2ZidCgF3QaB8GF3Lq3\nCMlUjJ24uigm929Gk5BSPPXDDnaecti9RoRwOJ+vib2yYcbAlqHXfpMlywj0T5rCyjegSgt4Yh08\nMMupQh0k2O3K28OVGXlPtj02cxsnE9LNLkkIp/fzztNMWnaQXk0r8fy1NszItUD0tzA5HJaOgbJ1\n4NFl8OACqBhW9AUXAQl2Oyvr58nMR1qQqzVDvtnK5bRss0sSwmn9FZvACwt20bJ6EO/d3wSXqzfM\nsFph1wL4LBKWPAX+5WHQjzDkVwhtZV7RRUCCvRDUDPZj+uAI4hIzeGTmNtKypBukEPa253QSw2ZF\nEVrGh2lXN/bSGg78Bl+2g8VDwc0b+s+FoSuhZudi08+lMEmwF5LIakF8OqAZu+ISGT47miyLPMAk\nhL0cu5jGkG+24u/lxqxHWxDo424E+pGVML0z/DAQLJlw3wwYvgHq3VUiAv1vEuyFqFvDCrxzXxPW\nH77I6Hkx5MoDTELY7FxSJg99tQWrhu+GtqRSKW9jg+iZd8PsPpAWDz2nwH+2QuP7waXkxZwsdyxk\nD0RUISk9hwm/7yfA2523eze6tcebhRBXXE7LZtCMLSSmZ/PDsNbUzDkCs8cbj/77loM7J0Hzh8HN\nxta8Ds5uwa6UcgWigNNa6x72GtcZPN6hBpfSs/liTSxBvu680K2e2SUJ4XDSsiw8MnMbJy6lM693\nII03/gf2LwHv0tDlDWgxDDx8zC6zWLDnFfvTwH4gwI5jOo0x3eqSmJ7DZ6tjKe3jwdD2NcwuSQiH\nkWXJZfjsaBLjDrKm9moqLVkCHn5w20vQeiR4Sevsq9kl2JVSIcDdwARgtD3GdDZKKd66txHJGTm8\n9dt+Arzc7buZrhBOypJr5Y3Zf3Lnsan091qHyxl3aDMK2j4DvmXMLq9YstcV+8fAGMDfTuM5JVcX\nxYf9wkjOzOHFxbtwd1P0bhZidllCFFu5KRfY8PXLvH7pZ9zcNS4Rj0CH58FfWnbciM23i5VSPYAL\nWuvom7xvmFIqSikVFR8fb+thHZanmyvTB0fQukYZnpsfw887T5tdkhDFT8ZlrCvexPJhE9pfWsTR\ninfi+tR2uPt9CfV8UFrbtgRPKTURGARYAC+MOfbFWuuHrveZiIgIHRUVZdNxHV16toVHvtlG1InL\nTO7fjLubFHDbLiGcSVYqbPkSvWkyKjOJJbmtuNzieQbf09XsyooFpVS01jriZu+z+Ypda/2y1jpE\na10N6A+sulGoC4OPhxtfD4kkPNRoGvbHnnNmlySEeXIy4a/P4ZMwWDWe/e6NuDNrIoc7fCqhXgAl\nb+V+MeLr6cY3j7QgLCSQUXO3s2LfebNLEqJo5eZA1DfwaTgsexldviFTa0/lrvj/0Om2zjzbpbbZ\nFTokuwa71nqNrGG/NX6ebsx8tAUNKgUycs52Vh+4YHZJQhQ+ay7EzIMpkfDrMxBQGT34F94MmsjE\n3f480aEGL3SrKw/zFZBcsRcDAV7uzHq0BXUr+PPEd9Es2yvTMsJJaQ37foEv2sKPw8DTDwbOx/rI\nMl7dFcQ3G4/zaNvqvHRnPQl1G0iwFxOB3u7MHtqSRpUDGDlnu6yWEc5Fazi8AqZ1hPmDwGqBvjNh\n2DosNbvy/KJdzN58kiduq8GrPepLqNtIesUUI4He7nz3WEuGfhvFM/N2kpGdS/8W19kNRghHcXwj\nrBoPJ/+CUqHQ63No0g9c3ci2WHl23k5+232W0V3rMKpzLQl1O5BgL2aMG6qRjJgdzUuLd5OWnctj\n7Rx/c11RAp2OhlVvQewq8KsAd38AzQaDmwcAmTm5jJyznVUHLvDK3fWlzYYdSbAXQ17urkwdFMHT\nP+xg/K/7yMi28GRnWR0gHMT5vbD6bTjwK3gHwR1vQeRQcPe+8pa0LAuPz4rir6MJTOjdiAdbVjWx\nYOcjwV5Mebi58OmAZrywcBfvLz9EWnYuY2SVgCjOEmKNQN+zCDz9oeNYaDUCvP7ZFzApI4dHZ25j\nx8nLfPhAmLTVKAQS7MWYm6sLH/QNw9vDlS/WxHIxJYu3+zTG3VXueYtiJPEUrHsPdswx+qC3ewba\nPAU+Qf/z1rNJGQz5ehtHL6by2cBw7mwsT1wXBgn2Ys7FRTHh3kaU9fNk8srDxKdm8dnAcHw95ZdO\nmCz1Aqz/AKK+Nl63eBzajTY2jb6Gg+dSGPLNVlIyLXz7SAva1CpbhMWWLJIODkApxeiudagY6MW4\nH3czYPpmvh4SSVm/kr1LjDBJ+iXYNBm2TAVLFjR7EDqMgVLXb0O9+WgCj8+KwtvdlflPtKZBJdm2\noTBJsDuQAS1CCfbz5Mm527nvi018+0gLqpX1NbssUVJkpRj9XP6aYvy88f3Q8WUoU/OGH/t11xlG\nz4shtIwPMx+JJKS07HJU2GSy1sF0aVCeuY+3Ijkjhz5fbGLnqUSzSxLOLicDNn1qNOha8zZU7wAj\nNsJ9X9001GdsOMaouTtoEhLIwuGtJdSLiAS7A2oWWppFI9rg6+nKgGmbpTOkKByWbNj2FUxuBstf\ngYphMHQV9J8D5Rve+KO5Vt5Yspfxv+7jjgblmT20JaV8PIqocCHB7qBqBPuxeERb6lTwZ/jsaKas\nOoytvfWFAIwGXTu/hykR8NtzULoaDPkdBv0IIc1v+vGkjBwembmNbzYeZ0ibanz+YHO83F0Lv25x\nhcyxO7Bgf0/mDWvFS4uMte4Hz6cy6f4m8odIFIzVCvt/MdaiXzwIFZvC3R9Crdshn89PHI1PZeis\nKE4mpDOxT2MGSEsMU0iwOzgvd1c+6teUOhX8mbTsICcS0pg2KIIKgV5mlyYchdZweLnx+P+5XRBc\nDx74Durfk+9AB1h/OJ7/zNmOq4ti9tCWtKohG02bRaZinIBSipEdazH1oeYcuZBKzykbiJGbqiI/\njq2Dr7vB9w9AVjL0ngYjNkGDnvkOda013246zpBvtlEx0JtfnmwnoW4yCXYnckfDCiwe2QYPNxce\nmPoXi6LjzC5JFFdxUfBtT/j2HkiKgx4fw5NRENYPXPI/lZeZk8vYH3fz+i976VQ3mEUj21AlSFa+\nmE2mYpxMvQoB/Pyftoycs53nFsQQdeISr9/TUObdheHcblg1AQ4tBZ+y0G0iRDwK7rc+dXciIY2R\nc7az90wyIzvW5Lk76uLqIr2MigObg10pVQWYBZQHNDBNa/2JreOKgivj58mcoS354M9DfLEmlphT\nSXzxUDhVy8jDTCXWxcPGTdG9i8ErEDq/Ci2HGzsYFcCyved4fkEMCvhqcARdGly7jYAwh7J1iZxS\nqiJQUWu9XSnlD0QD92qt913vMxEREToqKsqm44r8Wbn/PKPnx2DVmvf7htGtYQWzSxJFKfEkrHkX\nYr4HN2+j22KbUeBdqkDD5eRambTsINPWHaVx5UA+fzBcpl6KkFIqWmsdcbP32XzFrrU+C5zN+3mK\nUmo/UBm4brCLonN7/fL8OqodT36/nSe+i+bx9tUZ072edIh0dinnYN37ED0TlAu0HAHtngW/4AIP\neS4pk1Fzt7Pt+GUGtarKKz3q4+kmU3zFkc1X7P8YTKlqwDqgkdY6+V/fGwYMAwgNDW1+4sQJux1X\n3FyWJZcJv+1n1l8naFqlFB/3ayp9ZpxR+iXY8BFsnQ7WHGj2kNGgK7CyTcMu33uOlxbvJjMnl4l9\nGtOrqW3jiYLJ7xW73YJdKeUHrAUmaK0X3+i9MhVjnt92neXlxbuwWDWv9mhA/8gqsnmHM8hMhr8+\nM35kpxp7inZ8EYJs224uLcvCm0v2MS/qFA0qBjB5QFNqlfO3U9HiVhXZVEzewdyBRcCcm4W6MNfd\nTSoSXrUUzy+I4eXFu1m5/wLv3NdYWgA7qux02DoNNn4MGZeNh4o6jYNy9W0eOvrEZUbP38nJS+mM\n6FiTZ7vUwcNNpvAcgT1unirgW+CS1vqZ/HxGrtjNZ7Vqvt54jPeWHSTAy41372vC7fVlZYPDsGRB\n9Lew/n1IPQ+1ukDnV6BSM5uHzsm18unKw0xZfYSKgd581K8pLar/725IougV2VSMUqodsB7YDVjz\nvjxWa/379T4jwV58HDyXwtM/7ODAuRT6R1bh5bvqE+jtbnZZ4npyLRAzF9a+C0mnoGpbY+li1dZ2\nGf7AuWTGLNzFrrgk+oRX5r89GxLgJb8fiosin2O/FRLsxUuWJZcPlx9i+vqjlPXz5M1eDeneSPai\nLFasVmMN+pqJkHAEKoUbV+g1O99SP5fryczJZcqqI3y5NhZ/Lzcm9G7MXbIfabEjwS5uWcypRF5a\nvJv9Z5O5o0F53uzVSJqJmU1rOLgUVk+A83ugXANjDr3e3XYJdDC2rRu7eDdHL6bRp1llXunRgCBf\n6Z1eHEmwiwLJybUyY8MxPvrzEO6uLrzYvS4PtqyKizwqXrS0hqNrYNV4OB1trG7pNA4a9gEX+9zA\nTErPYeLS/fyw7RRVgrx5u3dj2tcu+Dp3Ufgk2IVNTiSkMfbH3Ww8kkB4aCne6NmIxiGBZpdVMpzc\nYgT68fUQEAK3jYGmA8HVPnPdVqvmp52nefv3A1xOz2Zou+o806UO3h7ysFFxJ8EubKa1ZtH200z8\nfT+X0rO5PzyEF7rVpVyATM8UirMxRk/0w8vBNxjaPw8Rj4Cb/ZaiRp+4zJu/7iPmVCJhIYFM6N2Y\nRpXlL2xHIcEu7CY5M4cpq47wzcZjeLi6MLJTLR5rV106RtpL/EFjDn3fz+BVCto9Ay2GgYf9ngw+\nk5jBu38c4OedZyjn78mL3evRu1llmWJzMBLswu6OXUzj7d/38+e+84SU9mbsXfW5s1EFeXK1oC4f\nNxp07foB3H2g9X+MH172u4JOz7Ywde1Rpq6LxaphWPsajOhYE19P6djtiCTYRaHZeOQiby7Zx8Hz\nKYSFBPJs1zrcVidYAj6/ks/AukmwfRa4uEHkUKNBl29Zux0iMyeXOVtO8sWaWC6mZnF3k4q8fGc9\nQkpLJ0ZHJsEuCpUl18qi7XFMXnmE04kZNK9amtFd69CmZhkJ+OtJu2g06Nr2FVgtEP4wdHgBAuy3\nXjzLksu8baf4bPURzidn0aZmGZ67ow7Nq8qTo85Agl0UiWyLlQXRp5iy6ghnkzJpUT2I0V3ryJ6X\nV8tMgk1TYPPnkJMOTfobDbpKV7PbIbItVhZGxzFl1WHOJGUSWa00o7vWpXVN+XVwJhLsokhl5vz/\nleKFlCwiq5XmsXY16NqgfMndLi07DbZMhY2fQGYiNLgXOo2F4Lp2O0RSRg7ztp1k5sbjnEnKpFlo\nKZ7rWpe2teRfTs5Igl2YIjMnl7lbTzJjwzHiLmcQGuTDo22r0TeiSsm5YZeTCdHfwPoPIC0eaneD\nzuOgYpjdDnHqUjpfbzzG/G2nSMvOpVWNIJ64rSYd5V6HU5NgF6ay5Fr5c995vtpwjOgTl/H3cmNg\ny1AGt65G5VLeZpdXOHJzYOccWPseJJ+Gau2NBl2hLe0yvNaaqBOX+XrDMZbtPYeLUvQMq8Sj7arL\nWvQSQoJdFBvbT15mxvpjLN1zFg20q1WWvhFVuKNBeedYC2/NhT2LjAZdl45C5Qi4/VWo0dEuw19I\nyWTx9tMsiDpFbHwaAV5uPNiqKg+3ria9fEoYCXZR7MRdTmdBVBwLo+M4nZhBgJcb9zarTN/mVWhU\nOcDxphC0hgO/wqoJEL8fyjcyOi7W6W5zg66cXCurDlxgQdQpVh+MJ9eqiahamr4RIfRoUqnkTGuJ\nf5BgF8WW1arZFJvAguhTLN1zjmyLlTrl/ejeqCLdG1agfkX/4h3yWkPsSuPx/zM7oEwt46Zog942\nNejKtljZFHuRZXvPs3zvORLSsinn70mf8BD6RoRQM9jPjichHJEEu3AISek5/BJzmiUxZ9l24hJa\nQ5Ugb7o3rEC3hhUIDy1dvB57P7EJVo6Hk5sgMNRYttikP7gW7Ao6LcvC2kPxLNt7jlX7L5CSZcHX\nw5WO9cpxX3hlOtQOxs1VtqMTBgl24XDiU7JYsf88y/aeY+ORi+Tkasr6edC6Zlna1CxD6xplqFrG\nx5yr+dPbjSv02JXgV954sCh88C036LLkWtl1Oom/YhPYfDSBrccukWWxUtrHna4NytOtYQXa1irr\nHPcehN0VabArpboDnwCuwFda63du9H4JdnEzyZk5rD5wgVUHLvBXbAIXUrIAqBjoResaZWhVswxN\nq5SiRlnfwr2ivbDfCPQDv4J3aePR/8jHwSN/j+anZVk4cC6Z6BOX2RSbwLZjl0jLzgWgbnl/2tQq\nwx0NKhBZrbRcmYubKso9T12BQ0BXIA7YBgzQWu+73mck2MWt0FoTG5/GX0cT2Jx3pZuQlg2Ap5sL\n9Sr406BSIA0rBdCwUgA1y/nZvk9nQiyseQd2LwAPP2jzJLQaCV4B13y71aqJT83i4LkU9p5JZu+Z\nJPadTebYxTT+/iNWM9iX1jXL0LpGWVrVCKKMn/3a8YqSoSiDvTXwX611t7zXLwNorSde7zMS7MIW\nVqsmNj6VPWeS2Hs6+UqQJmdarrwnwMuNKkE+hJT2pkpp478VAr0I8HInwNudAC93Ar3d8fNy++eT\nsUlxxjr0HbPB1YPcyMdJDB9JEv4kZeSQnGkhMT2bs0mZnLqUzqnLGcRdTifucgbZFuuVYSqX8s77\niyaQBpVKPUTJAAAUxElEQVQCCAsJlD72wmb5DXZ7rJmqDJy66nUcYJ8nMoS4BhcXRe3y/tQu70/v\nZsbXtNacTsxg75lkTiSkEXc5g1OX0omNT2PtoXgyc6zXHc/b3ZUyKonH1U/0509csDKfLnyRcy9x\nqwNh9fZrfq60jzshpX2oV8GfLvXLU6W0NzXL+dGwYiCBPvbZ7UiIgiiyxbBKqWHAMIDQ0NCiOqwo\nIZRShJT2uWZbWq01F1OzuZiaZVx15115J2XkkJWSQNjJWUSen4+bNZtdZe9kfaVHSfasyN1K4e/l\nRoC3cXVvXO27EeDlToVAL/xtne4RopDYI9hPA1Wueh2S97V/0FpPA6aBMRVjh+MKkS9KKYL9PQn2\nv2pOOysVtnwBOz+FrCRodB90HEvTsrVoal6pQtiFPYJ9G1BbKVUdI9D7AwPtMK4Q9peTCVEzYP2H\nkH4R6t4FncZBhUZmVyaE3dgc7Fpri1LqSWAZxnLHr7XWe22uTAh7ys2BHd/B2kmQcsbo49L5VQi5\n6X0oIRyOXebYtda/A7/bYywh7MqaayxZXDPR2GO0SkvoMw2qtze7MiEKjXQSEs7JaoX9v8Dqt+Hi\nQajQGAbOh9p32NygS4jiToJdOBet4cgKWDUezsZA2TrQdybU72VTgy4hHIkEu3AexzcYDbpObYZS\nVeHeL6HJA+AifVdEySLBLhxfXLRxhX50NfhXhLs/gGaDwc3D7MqEMIUEu3Bc5/YYc+gHfwOfMnDH\nWxA5FNyddOs9IfJJgl04noRYI9D3LAJPf2MdeqsRxs+FEBLswoEknoS178LOuUYf9HbPQJunwCfI\n7MqEKFYk2EXxl3Ie1r8P0TON1y2GQfvR4FfO1LKEKK4k2EXxlX4JNn4MW6ZBbjY0ewhuGwOBIWZX\nJkSxJsEuip/MZNj8Ofz1GWSlQOO+0PElKFPT7MqEcAgS7KL4yMmArdNhw0eQcQnq9TBujJZvYHZl\nQjgUCXZhPks2bP8W1r0Pqeeg5u3Q+RWoHG52ZUI4JAl2YZ5cC+yaB2vfMVa8hLaBvt9A1TZmVyaE\nQ5NgF0XPaoV9Pxlr0RMOQ8Wm0OMj40pdGnQJYTMJdlF0tIZDy2DVW3B+NwTXh36zjbl0CXQh7EaC\nXRSNo2uNQI/bCqWrQZ/pxnZ00qBLCLuTYBeF69Q2WPUmHFsHAZWhx8fGenRX2QhaiMIiwS4Kx9ld\nsHoCHPoDfMpCt4kQ8Si4e5ldmRBOz6ZgV0pNAu4BsoFY4BGtdaI9ChMOKv4QrHkb9v4IXoHGvqIt\nh4Onn9mVCVFi2HrF/ifwct6G1u8CLwMv2l6WcDiXTxgNumLmgps3tH8e2owC71JmVyZEiWNTsGut\nl1/1cjNwv23lCIeTfDavQde3oFyg5Qho9yz4BZtdmRAllj3n2B8F5l3vm0qpYcAwgNDQUDseVpgi\nLQE2fmS0ALBaoNkg6PACBFY2uzIhSrybBrtSagVQ4RrfGqe1/jnvPeMACzDneuNoracB0wAiIiJ0\ngaoV5stMMppz/fU5ZKdCk35Gg66g6mZXJoTIc9Ng11p3udH3lVJDgB7A7VprCWxnlZ0GW6fBho8h\nMxHq9zQadJWrZ3ZlQoh/sXVVTHdgDHCb1jrdPiWJYsWSZWxwse59SLsAtboaDboqNTW7MiHEddg6\nxz4F8AT+VMYj4Zu11sNtrkqYL9cCMd/D2vcg6RRUbQf9voPQVmZXJoS4CVtXxdSyVyGimLBaYe9i\no0HXpVio3Bx6ToYanaSfixAOQp48FQat4eBSo5/Lhb1QriH0nwt175RAF8LBSLCXdFrD0TVGoJ+O\ngqCacN8MaNgHXFzMrk4IUQAS7CXZyc2wcjyc2ACBVaDnpxA2EFzlt4UQjkz+BJdEZ3YaV+hH/gTf\ncnDnJGj+MLh5ml2ZEMIOJNhLkgsHjI6L+38B79LQ5Q1oMQw8fMyuTAhhRxLsJcGlY7DmHdg9H9x9\n4baXoPVIo/uiEMLpSLA7s6TTsG4S7PgOXNyg9ZPQ9hnwLWN2ZUKIQiTB7ozSLsL6D2HbV6Ct0HyI\n0UY3oKLZlQkhioAEuzPJSIRNn8LmL8CSAWED4LYXoXRVsysTQhQhCXZnkJUKW76ETZON7osNe0PH\nsRBcx+zKhBAmkGB3ZDmZEPU1bPgQ0uKhTnej42LFJmZXJoQwkQS7I8rNgR2zjRujyaehegfo/D1U\naWF2ZUKIYkCC3ZFYc2H3QlgzES4fg5BIuPcLqHGb2ZUJIYoRCXZHoDXsX2J0XIzfD+Ubw4B5UKeb\nNOgSQvwPCfbiTGs4shJWjYezO6FMbbj/G2hwrzToEkJclwR7cXV8oxHoJ/+CUqHQ63Njf1Fp0CWE\nuAlJieLmdLTRoCt2FfhVgLveh/CHwc3D7MqEEA7CLsGulHoOeB8I1lpftMeYJc75fUaDrgO/gncQ\ndB0PkUOlQZcQ4pbZHOxKqSrAHcBJ28spgRJijVUuuxeCp7/xYFGrEeAVYHZlQggHZY8r9o+AMcDP\ndhir5EiKMzaK3jEbXD2g7dPGD58gsysT4pbk5OQQFxdHZmam2aU4DS8vL0JCQnB3dy/Q520KdqVU\nL+C01jpGybK7/Em9YDToipphvI4cCu2fA//y5tYlRAHFxcXh7+9PtWrVkBywndaahIQE4uLiqF69\neoHGuGmwK6VWABWu8a1xwFiMaZibUkoNA4YBhIaG3kKJTiLjMmycbPR0sWRB04FGg65SVcyuTAib\nZGZmSqjbkVKKMmXKEB8fX+AxbhrsWusu1zl4Y6A68PfVegiwXSnVQmt97hrjTAOmAUREROgCV+xo\nslJg85dG18WsZGh0H3QaC2Vqml2ZEHYjoW5ftv7/LPBUjNZ6N1DuqkKOAxGyKiZPTgZsm2E06EpP\ngLp3Q+dxUL6h2ZUJUWKkpqbSsWNHLl26xIYNG6hUqdKV7z344INERUXh7u5OixYtmDp1aoHntIsb\neXzR3izZRqBPbgbLx0GFxjB0JQz4XkJdiCJksVh44IEHGDRoEJMmTaJXr14kJydf+f6DDz7IgQMH\n2L17NxkZGXz11VcmVmtfdntASWtdzV5jOSRrLuyabyxdTDwBVVpCn+lQvb3ZlQnh1LZt28Zjjz3G\n1q1byc3NpUWLFsybN4+PPvqIO++8k1GjRgHg6upK//79+fnnn3F3d+euu+66MkaLFi2Ii4sz6xTs\nTmld9NPdEREROioqqsiPWyisVtj/s9Gg6+IhqBgGnV+FWl2kQZcoEfbv30/9+vUBeGPJXvadSb7J\nJ25Ng0oBvH7Pjf+1+8orr5CZmUlGRgYhISG8/PLL+R4/JyeHli1b8sknn9C+ffG5ELv6/+vflFLR\nWuuIm31WWgoUlNZweLnx+P+5XVC2LjwwC+r3lEAXooi99tprREZG4uXlxeTJk2/psyNHjqRDhw7F\nKtRtJcFeEMfWGYF+aguUrga9p0LjvuDianZlQpjqZlfWhSUhIYHU1FRycnLIzMzE19c3X5974403\niI+PZ+rUqYVcYdGSYL8VcVGw8k04thb8K0GPj6DZIHB1jjvpQjiqJ554gvHjx3Ps2DFefPFFpkyZ\nctPPfPXVVyxbtoyVK1fi4mRtsCXY8+PcHuMK/dBS8CkL3d6GiEfB3dvsyoQo8WbNmoW7uzsDBw4k\nNzeXNm3asGrVKjp37nzDzw0fPpyqVavSunVrAPr06cNrr71WFCUXOgn2G7l42LgpuncxeAZC51eg\n5Qjw9DO7MiFEnsGDBzN48GDAWPmyZcuWfH3OYrEUZlmmkmC/lsSTsOZdiPke3LyNXi5tRoF3abMr\nE0KIm5Jgv1rKOVj3PkTPBOUCLYdDu9HgF2x2ZUIIkW8S7ADpl2DDR7B1OlhzoNlD0OEFCAwxuzIh\nhLhlJTvYM5Nh8+ewaQpkp0KTB6DjSxBUw+zKhBCiwEpmsGenw7bpsOFjyLgE9e+BTuOgXP2bf1YI\nIYq5khXslmzY/q0xj556znjsv/MrUKmZ2ZUJIYTdONeq/OvJtRhb0H3aHH5/3uiF/shSeGiRhLoQ\nTua///0v77//fr7f/8svv/DOO+8U6Fg//fQT+/btu/L6tddeY8WKFQUay56c+4rdaoV9P8LqiZBw\n2Ajxez6Gmp2ln4sQAovFQs+ePenZs2eBPv/TTz/Ro0cPGjRoAMCbb75pz/IKzDmv2LWGg0thagdY\n+Ci4uEG/2fD4aqh1u4S6EE5mwoQJ1KlTh3bt2nHw4EEAYmNj6d69O82bN6d9+/YcOHAAgCFDhjB8\n+HBatmzJmDFjmDlzJk8++SRJSUlUrVoVq9UKQFpaGlWqVCEnJ4fp06cTGRlJWFgY9913H+np6Wza\ntIlffvmFF154gaZNmxIbG8uQIUNYuHAhf/zxB3379r1S35o1a+jRowcAy5cvp3Xr1oSHh9O3b19S\nU1Pt/v/D+a7Yj64xHv+P2walqxs90RvdJw26hCgKS1+Cc7vtO2aFxnDn9adKoqOj+eGHH9i5cycW\ni4Xw8HCaN2/OsGHD+PLLL6lduzZbtmxh5MiRrFq1CjA24N60aROurq7MnDkTgMDAQJo2bcratWvp\n1KkTv/76K926dcPd3Z0+ffrw+OOPA0aL4BkzZjBq1Ch69uxJjx49uP/++/9RU5cuXRg2bBhpaWn4\n+voyb948+vfvz8WLF3nrrbdYsWIFvr6+vPvuu3z44Yd2b2XgPMF+aqvRoOv4egioDPd8Ak0flAZd\nQji59evX07t3b3x8fADo2bMnmZmZbNq06R9XzVlZWVd+3rdvX1xd//dir1+/fsybN49OnTrxww8/\nMHLkSAD27NnDK6+8QmJiIqmpqXTr1u2GNbm5udG9e3eWLFnC/fffz2+//cZ7773H2rVr2bdvH23b\ntgUgOzv7Sq8ae7I52JVSo4D/ALnAb1rrMTZXdSvO7jKu0A8vA99g6P4ONH8E3L2KtAwhBDe8si5K\nVquVUqVKsXPnzmt+/3ptfXv27MnYsWO5dOkS0dHRVxqJDRkyhJ9++omwsDBmzpzJmjVrblpD//79\nmTJlCkFBQURERODv74/Wmq5duzJ37twCn1t+2DTHrpTqBPQCwrTWDYH834q2VfwhmP8wTG1v9EW/\n/XV4OgZajZBQF6IE6dChAz/99BMZGRmkpKSwZMkSfHx8qF69OgsWLABAa01MTMxNx/Lz8yMyMpKn\nn36aHj16XLmqT0lJoWLFiuTk5DBnzpwr7/f39yclJeWaY912221s376d6dOn079/fwBatWrFxo0b\nOXLkCGDM4x86dMim878WW2+ejgDe0VpnAWitL9he0k1cPg4/joDPW8KRFdBhjBHo7UeDR/6a6wsh\nnEd4eDj9+vUjLCyMO++8k8jISADmzJnDjBkzCAsLo2HDhvz888/5Gq9fv37Mnj2bfv36Xfna+PHj\nadmyJW3btqVevXpXvt6/f38mTZpEs2bNiI2N/cc4rq6u9OjRg6VLl165cRocHMzMmTMZMGAATZo0\noXXr1ldu6tqTTXueKqV2Aj8D3YFM4Hmt9babfa7Ae56unQRr3zVuhEYOhXbPgm/ZWx9HCGE319qb\nU9iuUPc8VUqtACpc41vj8j4fBLQCIoH5Sqka+hp/WyilhgHDAEJDQ2922GsrFQrhg6HD8xBQqWBj\nCCGEk7tpsGutu1zve0qpEcDivCDfqpSyAmWB+GuMMw2YBsYVe4GqDetn/BBCCHFdts6x/wR0AlBK\n1QE8gIu2FiWEEKLgbF3u+DXwtVJqD5ANPHytaRghhHPTWqPkiW67sTVGbQp2rXU28JBNFQghHJqX\nlxcJCQmUKVNGwt0OtNYkJCTg5VXwZdvO8+SpEMIUISEhxMXFER//P7fWRAF5eXkRElLwHdwk2IUQ\nNnF3d6d69epmlyGu4pzdHYUQogSTYBdCCCcjwS6EEE7GppYCBT6oUvHAiQJ+vCzOs1ZezqX4cZbz\nADmX4sqWc6mqtQ6+2ZtMCXZbKKWi8tMrwRHIuRQ/znIeIOdSXBXFuchUjBBCOBkJdiGEcDKOGOzT\nzC7AjuRcih9nOQ+QcymuCv1cHG6OXQghxI054hW7EEKIG3DIYFdKjVdK7VJK7VRKLVdKOeyuG0qp\nSUqpA3nn86NSqpTZNRWEUqqvUmqvUsqqlHLI1QtKqe5KqYNKqSNKqZfMrqeglFJfK6Uu5HVddVhK\nqSpKqdVKqX15v7eeNrumglJKeSmltiqlYvLO5Y1CPZ4jTsUopQK01sl5P38KaKC1Hm5yWQWilLoD\nWKW1tiil3gXQWr9oclm3TClVH7ACUzG2SCzA3ofmUUq5AoeArkAcsA0YoLXeZ2phBaCU6gCkArO0\n1o3MrqeglFIVgYpa6+1KKX8gGrjXQX9NFOCrtU5VSrkDG4CntdabC+N4DnnF/neo5/EFHO9vpzxa\n6+Vaa0vey81AwVu6mUhrvV9rfdDsOmzQAjiitT6a1476B6CXyTUViNZ6HXDJ7DpspbU+q7Xenvfz\nFGA/UNncqgpGG1LzXrrn/Si03HLIYAdQSk1QSp0CHgReM7seO3kUWGp2ESVUZeDUVa/jcNAQcUZK\nqWpAM2CLuZUUnFLKVSm1E7gA/Km1LrRzKbbBrpRaoZTac40fvQC01uO01lWAOcCT5lZ7Yzc7l7z3\njAMsGOdTLOXnPISwN6WUH7AIeOZf/1p3KFrrXK11U4x/lbdQShXaNFmx7cd+o020/2UO8DvweiGW\nY5ObnYtSagjQA7i9OG8teAu/Jo7oNFDlqtcheV8TJsqbj14EzNFaLza7HnvQWicqpVYD3YFCucFd\nbK/Yb0QpVfuql72AA2bVYiulVHdgDNBTa51udj0l2DagtlKqulLKA+gP/GJyTSVa3g3HGcB+rfWH\nZtdjC6VU8N8r3pRS3hg36Qsttxx1VcwioC7GKowTwHCttUNeXSmljgCeQELelzY74gofpVRv4FMg\nGEgEdmqtu5lb1a1RSt0FfAy4Al9rrSeYXFKBKKXmAh0xugieB17XWs8wtagCUEq1A9YDuzH+rAOM\n1Vr/bl5VBaOUagJ8i/F7ywWYr7V+s9CO54jBLoQQ4voccipGCCHE9UmwCyGEk5FgF0IIJyPBLoQQ\nTkaCXQghnIwEuxBCOBkJdiGEcDIS7EII4WT+D0JMm6YIdH6LAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8e8d462a50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt",
    "\n",
    "%matplotlib inline",
    "\n",
    "\n",
    "x = np.linspace(-3, 3)",
    "\n",
    "x_squared, x_squared_der = s.run([scalar_squared, derivative],",
    "\n",
    "                                 {my_scalar: x})",
    "\n",
    "\n",
    "plt.plot(x, x_squared, label=\"x^2\")",
    "\n",
    "plt.plot(x, x_squared_der, label=\"derivative\")",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why that rocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "my_vector = tf.placeholder('float32', [None])",
    "\n",
    "\n",
    "# Compute the gradient of the next weird function over my_scalar and my_vector",
    "\n",
    "# warning! Trying to understand the meaning of that function may result in permanent brain damage",
    "\n",
    "\n",
    "weird_psychotic_function = tf.reduce_mean((my_vector+my_scalar)**(1+tf.nn.moments(my_vector, [0])[1]) + 1. / tf.atan(my_scalar))/(my_scalar**2 + 1) + 0.01*tf.sin(",
    "\n",
    "    2*my_scalar**1.5)*(tf.reduce_sum(my_vector) * my_scalar**2)*tf.exp((my_scalar-4)**2)/(1+tf.exp((my_scalar-4)**2))*(1.-(tf.exp(-(my_scalar-4)**2))/(1+tf.exp(-(my_scalar-4)**2)))**2",
    "\n",
    "\n",
    "\n",
    "der_by_scalar, der_by_vector = <student.compute_grad_over_scalar_and_vector() >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x7f8e84b43f90>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD8CAYAAAB0IB+mAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNXZwPHfmcmeyUoWlgAJ+yZbcEEWwbpgtW5Vwa31\ntRVt69L6qtXat9q3Sm1d+tZaRa1brYo7Iu5UQhEEZAdZZAsQCEtC9mSynvePMxNCyDL7TCbP9/PJ\n5yYzd+49J4Fnzjz33OcorTVCCCHCnyXYDRBCCBEYEvCFEKKbkIAvhBDdhAR8IYToJiTgCyFENyEB\nXwghugkJ+EII0U1IwBdCiG5CAr4QQnQTEcFuQEtpaWk6Ozvb49dXVVURHx/vuwYFSbj0A6QvoShc\n+gHSF6c1a9YUaa3TO9svpAJ+dnY2q1ev9vj1eXl5TJs2zXcNCpJw6QdIX0JRuPQDpC9OSqm9ruwn\nKR0hhOgmJOALIUQ3IQFfCCG6iZDK4belvr6egoIC7HZ7p/smJSWxdevWALTKvwLRj5iYGLKysoiM\njPTreYQQoSPkA35BQQEJCQlkZ2ejlOpw34qKChISEgLUMv/xdz+01hQXF1NQUEBOTo7fziOECC0h\nn9Kx2+306NGj02AvXKeUokePHi59ahJChI+QD/iABHs/kN+pEN1Plwj4QggR1lY8Q/qRZX4/jQR8\nFz355JMMHz6ca6+91ifHy8/P5/XXX2/+efXq1dx+++0+ObYQootZ8QxpRSv9fpqQv2gbKp5++mkW\nLVpEVlaWT47nDPjXXHMNABMmTGDChAk+ObYQoouxl1Ifb/P7aWSE74JbbrmF3bt3c8EFF5CUlMRj\njz3W/NyoUaPIz88nPz+f4cOHc9NNNzFy5EjOO+88ampqANi5cyfnnHMOY8aMYfz48ezatYt7772X\npUuXMnbsWP7yl7+Ql5fHRRddBMCxY8e49NJLGT16NGeccQYbN24E4MEHH+TGG29k2rRpDBgwgCef\nfDLwvwwhurpFv4cFIfRpuqkJ7OU0RPi/JlCXGuH//sNv2XKwvN3nGxsbsVqtbh1zRO9EHvjByA73\nmTt3Lp9++imLFy/mqaeeane/HTt28MYbb/D8889z1VVX8e6773Lddddx7bXXcu+993LZZZdht9tp\namrikUce4bHHHmPhwoWAqaPhNGfOHMaNG8f8+fP58ssv+dGPfsT69esB2LZtG4sXL6aiooKhQ4fy\ns5/9TObSC+EqrWHdq1B1FIbMgGHfD3aLoLYc0DREyAi/S8nJyWHs2LEA5Obmkp+fT0VFBQcOHOCy\nyy4DzA1PcXFxHR5nxYoVXH/99QCcffbZFBcXU15u3uguvPBCoqOjSUtLIyMjg8OHD/uxR0KEmWO7\nTbBXVvjkHqirCnaLwF4KEJCA36VG+J2NxANx41VERARNTU3NP7ecyx4dHd38vdVqbU7p+FLrczQ0\nNPj8HEKErf2OC6MXPgYLfwVL/gzn/j64baoxAb8+Ukb4ISc7O5u1a9cCsHbtWvbs2dPh/gkJCWRl\nZTF//nwAamtrqa6uJiEhgYqKijZfM3HiRF577TXApHrS0tJITEz0YS+E6Kb2rYCYJBh/A4y7Dr5+\nCg5vCW6bmkf4/s/hS8B30w9/+EOOHTvGyJEjeeqppxgyZEinr3n11Vd58sknGT16NGeeeSaHDh1i\n9OjRWK1WxowZw1/+8pcT9r/vvvtYs2YNo0eP5t577+WVV17xV3eE6F72rYC+p4PFAuf8L0QnwBf/\nE9w21UhKJ+Tk5+c3f//555+3uc/mzZubv7/rrruavx88eDBffvnlSfu3fsy5+EFqamrzJ4KWHnzw\nwXbPJ4ToRPUxKNoOY2aan+N7wMjLYfO7wW2XjPCFEMLH9q8y275nHH8sNccE3JqS4LQJAjrCl4Av\nhOge9q8ASyT0GX/8seT+Zlvi0gqB/mEvBUsEjdYYv5/KJwFfKfWiUuqIUmpzi8ceVEodUEqtd3yF\nwIRXIUS3tW8l9BoDkbHHH0vJNtvSIAb8mlKISYYAFDT01Qj/ZWBGG4//RWs91vH1sY/OJYQQ7mmo\ng4Nrod8ZJz6e4hzh5we8Sc3spRCbHJBT+STga63/AxzzxbGEEMLnCjdAg93M0GkpJgliU4Kc0ikz\nI/wA8HcO/zal1EZHyifFz+cSQoi27V9htq1H+GDy+MEc4dcEboTvz2mZzwB/ALRj+zhwY+udlFKz\ngdkAmZmZJ9SUAbO+a3s3KLXW2Njo8r7emDNnDjabzeVyxh9//DHbtm3jzjvvdGn/lv1YuHAhgwYN\nYtiwYQA89NBDTJo0ienTp3vW+BbsdvtJv29fq6ys9Ps5AiVc+hIu/QDX+zJy80JsMT1ZuXoLcOKN\nViMa4rEd3MqqIP1OTisppKIhPiB/F78FfK11c5EXpdTzwMJ29nsOeA5gwoQJ2jkX3Wnr1q0ul0sI\n1Jq20dHRREdHu3SuhoYGZs6c6dbxW/bjs88+IzIyklNPPRWAP/3pT+43uB0xMTGMGzfOZ8drS15e\nHq3/pl1VuPQlXPoBbvRl469g4MS2961fDCu+YdrUKWBxr/iiT6ysJa7fEGw2m9//Ln5L6SilerX4\n8TKgS98l9PDDDzNkyBAmT57M9u3bAdi1axczZswgNzeXKVOmsG3bNgBuuOEGbrnlFk4//XTuuece\nXn75ZW699VbKysro379/cy2eqqoq+vbtS319Pc8//zynnnoqY8aM4brrrqO6uprly5ezYMEC7r77\nbsaOHcuuXbu44YYbeOedd/j000+58sorm9vXsrzy559/zsSJExk/fjxXXnkllZWVAf5tCRFCtIby\ng5DUzloWKf2hsQ4qCgPbLjBts5d1rZSOUuoNYBqQppQqAB4ApimlxmJSOvnAzV6f6JN74dCmdp+O\nbWwAq5td6nkKXPBIh7usWbOGefPmsX79ehoaGhg/fjy5ubnMnj2buXPnMnjwYFauXMnPf/7z5rtn\nCwoKWL58OVarlZdffhkw6amxY8eyZMkSpk+fzsKFCzn//POJjIzk8ssv56abbgLg7rvv5oUXXuC2\n227j4osv5qKLLuKKK644oU3nnHMOs2fPpqqqivj4eN58801mzZpFUVERDz30EIsWLSI+Pp4//elP\nPPHEE/zud79z7/ciRLioPmYu2Cb2aft559TMkr3tvyn4S20F6EZz0bbe/6fzScDXWl/dxsMv+OLY\noWDp0qVcdtllzWWNL774Yux2O8uXLz9hlF1bW9v8/ZVXXtlmbf6ZM2fy5ptvMn36dObNm8fPf/5z\nwJRJ+O1vf0tpaSkVFRXMmNHWLNfjIiIimDFjBh9++CFXXHEFH330EX/+859ZsmQJW7ZsYdKkSQDU\n1dUxceJEr38HQnRZ5QfMtr2An9xiamb2pIA0qZmjrAKxXSjgB0wnI/GaAOXwAZqamkhOTm5emKS1\n+Pi262JcfPHF/OY3v+HYsWOsWbOGs88+GzBpoPnz5zNmzBjmzp3LihUrOm3DrFmzeOqpp0hNTWXC\nhAkkJCSgtebcc8/ljTfe8LxzQoSTzgJ+Ul9QluDcfOUoq0BMMrS/tpPPSGkFF0ydOpX58+dTU1ND\nRUUFH374IXFxceTk5PD2228DoLVmw4YNnR7LZrNx6qmncscdd3DRRRc1fwqoqKigV69e1NfX89Zb\nbzXv31EZ5bPOOou1a9fy/PPPM2vWLADOOOMMli1bxs6dOwFzneC7777zqv9CdGnNAb93289HRJk3\ng2BMzWw5wg8ACfguGD9+PDNnzmTMmDFccMEFzTNmXnvtNV544QXGjBnDyJEj+eCDD1w63syZM/nX\nv/51wuydP/zhD5x++ulMmjSJwYMHNz8+a9YsHn30UcaNG8euXbtOOI7VauWiiy7ik08+ab5gm56e\nzssvv8zVV1/N6NGjmThxYvPFZCG6pfKDYIkAW0b7+6RkByngl5ltgG68QmsdMl+5ubm6tS1btpz0\nWHvKy8td3jeUBaof7vxuPbV48WK/nyNQwqUv4dIPrV3sy7uztX5iZMf7vP9zrR8d4pM2uWXNP7V+\nIFHrY/le/V2A1dqFGCsjfCFEeCs/0H7+3iklGyoPQb3vlyXtkKR0hBDCh8oPtJ+/d2qumrnP7805\nQU2puWAcFZjJJl0i4JtPLMKX5HcqugXnTVedBvwgVc20l5oCbpbAhOKQD/gxMTEUFxdLgPIhrTXF\nxcXExPh/wQUhgsp501VnN1S1vPkqkJy18AMk5OfhZ2VlUVBQwNGjRzvd1263h0UQC0Q/YmJiyMoK\n8F2FQgRaZ1MyneLTITIuOCP8AOXvoQsE/MjISHJyclzaNy8vz+/FwAIhXPohRNCVHzTbzi7aKhWc\nMskBHuGHfEpHCCE8Vl5gtp0FfDB5/EDfbRvgEb4EfCFE+HLlpiunpL5Qtt//bWqpxnHRNkAk4Ash\nwlfZAUjo5Vqd+/h0c+drYwCqmMHx0siS0hFCCB9wZQ6+U1yq2VYX+689LdVXQ1O9pHSEEMInXJmD\n7xSfZraBCvgtK2UGiAR8IUR40tq1sgpOcY6AX1Xkvza1FOCyCiABXwgRrmpKOl7pqrXmEX6AAn5X\nHeErpV5USh1RSm1u8ViqUuoLpdQOxzbFF+cSQgiXlDmnZLqaw+9httXH/NOe1rrwCP9loPWafPcC\n/9ZaDwb+7fhZCCECw3nTlavr1MY6LtoGKqXTVUf4Wuv/AK3fFi8BXnF8/wpwqS/OJYQQLil3c4Rv\njYDYlMCldJwj/ADOw1e+KkqmlMoGFmqtRzl+LtVaJzu+V0CJ8+dWr5sNzAbIzMzMnTdvnsdtqKys\nxGazefz6UBEu/QDpSygKl35Ax33J2f0q/fa9x5Kz3gHlwjx84LSVP6fSls2Wkff4spltyt7zGtl7\n3yLvrPdAWb36u0yfPn2N1npCpzu6skqKK19ANrC5xc+lrZ4v6ewYba145Y5wWcknXPqhtfQlFIVL\nP7TupC/v3az14yPcO+A/ztP6pQu9apPLPrpL6zl9m3/s6iteHVZK9QJwbI/48VxCCHGisgLX0zlO\n8WmBm4dvL4PYwKVzwL/TMhcAP3Z8/2PAtRW+hRDCF9y56coprkdgL9oG8IIt+G5a5hvA18BQpVSB\nUuonwCPAuUqpHcA5jp+FECIwKo+YOjruiOthRviBWHApwJUywUf18LXWV7fz1Pd8cXwhhHBLfQ3U\nVRy/mcpV8WmgGx3B2M+3DtWUQvpQ/56jFbnTVgjhGw21ULwLdv7bbIPJmZaJT3fvdc3lFQKQx7cH\ntjQydIEVr4QQXcD8X8D61wBHKiQlB25fZ1aSCoYqxxwRdwN+vPNu2yJgkE+bdBJ7ecADvozwhRDe\nqTwCG16HIefDpc/A5DuhZA8c2hi8Nnk8wncGfD+P8BvroaFGAr4QoovZ/C7oJjjn9zD2Gph4q7nR\nacuC4LWp6qjZ2jxN6fh5pk5thdlGJ/j3PK1IwBdCeGfjW9BzNGQMMz/H94DsSbBlfmBmu7TFGfA9\nHuH7O+CXm210on/P04oEfCGE54p2wsG1MPqqEx8fcQkU74QjW4PTrqoiiIyDqHj3XhcVZ17n74u2\ndmfAlxG+EKKr2PQWoGDUFSc+PuwH5vGtQUrrVB11f0qmU1wA7rZ1pnRiZIQvhOgKtIaNb0LOVEhs\ndYNTQib0mwhbgnSDfeUR99M5TvE9ApjSkRG+EKIrKFgNJfkwembbz4+4BI5sgaIdAW0WYFI6ngb8\nuLQAXrSVWTpCiK5g45sQEQPDf9D2887HgzHKrzrqRcDv4f9Vr+xlZisjfCFEyNMatn0Eg89rPw+d\n1AeyToWtHwa2bU1NJiXjcUonLQApHcnhCyG6irICqDgI2VM63q//JDj8rbnRKFDspdDU4N0Iv74a\n6qp9266WasvBEmE+IQWQBHwhhPsKVplt39M63i9jODTVw7E9/m+Tk6d32To5Z/f4c5RfW2Hm4Ae4\n9IQEfCGE+/avMvPVM0d1vJ+zGuTRAM7Hb77pytNpmQEor2AvD3j+HiTgCyE8sX8V9B5vFv7uSNoQ\nsz263f9tcvK0cJpTICpm1lYEPH8PEvCFEO6qrzGF0TpL54C50zW5Hxzd5v92OTlTOrYMz14fkJRO\necDLKkAAyiMrpfKBCqARaNCurKwuhAhdB9eZi6KuBHyA9GEBHuEfBRTEpnr2+jjH6/yZ0qkth8Q+\n/jt+OwI1wp+utR4rwV6IMLB/pdlmuRHwi3ZAY4P/2tRS1VETtDtLN7UnJtnMoPHnzVf24IzwJaUj\nhHDP/m8gdeDxxUI6kz4MGmuhdK9/2+XkzU1XYGbOxPm5vEJtRdhetNXAIqXUGqXU7ACcTwjhL1qb\nEb6r6RwwAR8CVznTm7IKTnFp/rtoq7VJ6QThom0gljicrLU+oJTKAL5QSm3TWv/H+aTjTWA2QGZm\nJnl5eR6fqLKy0qvXh4pw6QdIX0KRN/2IqSnkjOoittckU+jiMawN1UwBdq/6hH2HbR6dtz1t9eW0\no3uptOWwxYu/1Zg6C5bC3azzw9/b0ljL1KYGdh8oYl+L4wfk35fWOmBfwIPAXe09n5ubq72xePFi\nr14fKsKlH1pLX0KRV/1Y/4bWDyRqXbjJvdc9PkLrd37q+Xnb0WZf5vTV+qO7vTvwWz/W+snx3h2j\nPeWHzO9w5XMnPOzN3wVYrV2IwX5N6Sil4pVSCc7vgfOAzf48pxDCj/avgqgEcwetOzKGBWZqZkMt\n1Jb5KKXjpxx+c6XM8EvpZALvK3P7cATwutb6Uz+fUwjhL/tXQVYuWKzuvS59GOR/BU2N7r/WHc1l\nFTy8y9YproejJo8f2lvrqJQZbjl8rfVuYIw/zwFQWl3Hnz/bTqK9gRHldjISA1uQSIhuoa4ajnwL\nk+90/7XpQ6HBDqX7IDXH921z8nQt29Zik83WXnZ8Xr6vBGkBcwjMRVu/211UxYfrD1JR28DcDf9m\nQFo8pw9I5dTsVE7LSSUrJS7YTRSi6zu0CXQT9Bnv/mudM3WObvNzwPeycJpTbIrZ1pT4PuDbg7OA\nOYRJwB/fL4X1D5zHqx9+SX1KDl/vLmbhxkLeWLUfgF5JMUzITuXU7BRy+6cwrGciVktgq9QJ0eUV\nrjfb3uPcf21zTZ1tMPQC37WpNW8LpznFOEb4NaXeHactMsL3ntWiyE6yMm3qAG6aOoDGJs32QxWs\n2lPM6r0lrNpTzIcbDgJgi45gXL9kxvdLYXz/FMb1SyYxJjLIPRAixB1cB7ZMSOjV+b6txSZDQm//\nl1hwFk7ztI6OQ0N0EhFAfsEBqiyDSLNFk+mrVLFzPduYwC5vCGEU8FuzWhQjeicyonciN0zKQWtN\nQUkNa/aWsHrvMVbnl/C3L3fQpM2NdYMzbOYNoF8K4/snMyDNhkU+BQhx3MF10Gus5zXc04f6f6ZO\n1VGzqEiU5/P9j5Tb+f0H+fwdeGLBShY0WbFaFHMuG8XMU/t530YZ4fufUoq+qXH0TY3j0nGmaFGF\nvZ4N+8tYs7eEtftK+HhTIfO+MWmghJgIxvZNZlzfZMb2S2ZMVjI9bNHB7IIQwVNbaUbnIy71/Bjp\nQ2Htq+ZOU38t/OG8y9bD428qKOOmf64mokaBFW45PZXvD8zltZV7+fW7mzhYaueX5wxGedN+exlE\nxII18FmFbhPw25IQE8nkwWlMHmzyfU1Nmt1FVazbV8K6/aWs3VvCU4t30qTN/n1TYxmdlcyYrCRG\nZyUzsnciCZIKEt3BoU2A9ix/75TcH+qrzALhrtbhcVfVUY/z94u3HeFnr62hR3w0z80+B16AEclN\njBjVk+8Nz+C+9zbx13/v4FCZnT9efornGYAg1dGBbh7wW7NYFIMybAzKsHHlhL4AVNU2sOlAGev3\nl7KxoJT1+0r5aGNh82sGpMUzsk8SI3qZ9NHwXgmk26K9GwEIEWoOrjPb3mM9P0ayIx1Sto+aSHNR\nNDbKx3Pcq46a6wxuOlRm55dvrmdguo1XbjyNNFs0RMabufhApNXCo1eMJjMxmr8v3sXYfslcfZqH\n6Z0g1dEBCfidio+O4IwBPThjwPERSVFlLZsKyth0wHytyT/WfEEYICUuksGZCQzOsDEg3caAtHiy\n0+LpnRxDdIQfbzoRwl8OrjMXaxN6evTygpJq3ltby+3Abc98wIf1B4myWrhkbG9+MiWHYT19FACr\nijpfdrEVrTV3v7OBuoYmnrpmvAn2YC4015Q076eU4q7zhrI6v4RHPtnGuSMyj+/rDhnhdy1ptmim\nD8tg+rDjMwFKq+vYWljB1sJydhyp4LvDlXy44SDl9hNrgKcnRNMnOZaMhGjSE6JJs0WTHBdJYkwk\nibGRxEZa2VHSSI+CMiwWUCiUgiataWjUNDRp6hubqGtoorbBuW1s/r6+sYm6xiYaG3XzOZWCqAgL\n0RFWYiItzedKio0kMzGGHvFRcoFadOzgOo/SOUfK7Tz55Q7e/GY/STRxeyRckt3IsAFDOVBaw3tr\nC3h7TQFnDUnn8avGeBZAnbT2KKXz6oq9LN1RxEOXjiInLf74E7EpJ03LVErx8GWjuOCvS5nz0Vae\nmOnBJ54g1cIHCfg+kxwXxcSBPZg48PgnAa01JdX17CmqZE9RNQdKajhQWs3BUjt7i6tZvbeEY1V1\nbR9w5VcBajlEWBSZiTFkpcTSv0cc/XvEMyjDxuAMG/1S44iwyrIJ3Zq9HIp3wOir3HrZ17uKue2N\ntZTV1HPVhL78YtpAmJvIOb1qOWf6IADuOX8or63cx9++3MFVc7/m1Z+eTp/kWA/bWQaNdRDv+pTM\nnUcqmfPxVqYNTefa01ulaGJOHOE7DcpI4OapA3lq8U6umJDFmQPdvGZQW+H1tFFPScD3I6UUqfFR\npManktu/7bv1GhqbqLA3UG6vp7ymAXtDI6tWr2PoiFE0ao3W5o3DYlFEWBRWiyLKaiE60kKU1UpU\nhIWYSDN6j7QqoiIsRFotWC0K55i9SUOd41NBTX0j5TX1lNXUU1pdx+HyWg6X2ykss7PvWDVfbjtK\nUWVBc/uirBYGZdgY1jOBYb0SGNYzkeG9EklPkBlL3cahjWbr4ghfa80LX+3hj59so3+PON646QwG\nZzpSGEl9oWx/877JcVH8YvogTstJ5caXv+GKZ5bz6k9OZ1CGB9Mq3bzLVmvNfe9tJCbSyp9/OPrk\n626xyXBsd5uvvfXsQSzYcJDfzt/MJ3dMcS9VW1sRlDn4IAE/6CKsFlLio0iJj2p+rCrfyrQR7l94\n6khUhAUcMbqzEVRlbQO7jlSy40glOw5XsO1QBct2FfHeugPN+6TZohjiuE4xODOBgek2ctLiyUiI\nlvRQuHFesO3VefqitqGRX7+zkfnrD3LeiEwev2rMiTPZkvuZejqtnJqdypuzJ/KjF1dx1bNf897P\nziS7ZXrFFW7eZfvhxkK+yS/hj5ef0nb9rdi2R/gAMZFWfn/JSP7rpW94a3UB15/R3/V21pZLDl+E\nDlt0BGP6JjOmb/IJjx+rqmPboXK2FVaw7VA52w9X8s6aAqrqGpv3iYm0kJUSR6+kGHomxlBTUscO\ny26S4yJJiIkkJtJCbKT5ZKKU+RSiMZ906hvN9Ql7fSM19Y3Y6xuprnN+NZhtbSPVjufs9Y3U1ptr\nFg1NTTQ0Oj4RYa5fWJRq/rTjPG9slJX4qAhsMREkxESSHBtJSnwkKXFRpMZHkWaLpoctKrgX17WG\n+mqwl6FjUyiyWzhQWsPhcjvFlXUUVdZSWdtAfaPps9WiHO2PJCMxhhG9EslKifXdTLGD6yAxC2wd\nj5zLauq5+dXVrNh9jDvPHcKt0wed/Oaf3Bf2Lmvz9SN6J/L2LRO57Oll3PzqGt7/xZnERbkRotwo\nnFZT18gfP97KyN6JXOWYkXeSNnL4LU0bkk5u/xSeWbyTmRP6mkFVZ5qaHBdtJYcvQlxqfBRnDkw7\nIWepteZAaQ17iqrIL64mv6iKg6U1FJbZ+e7wUYoq6lm42/ul7awWRZwjYMdFWYmJdH5ZSIiMINJq\nIcKisLQIck36+EXu2oYmiqvqqC5ppKq2gQp7A5W17S+qnRgTQXpCNBkJMY5tNBVH6ylJKiDNFt38\nxpASF0WkF9c46hubTBA/sJPIzW+Tkf8BCTX7idCmbVU6lrzGU/mgaRLLm0bS5FjCIjbSSoTVpPka\nGjUVrfqSFBvJ6Kwkpg/N4NwRmfRN9aKA4MF1nU7HPFBaww0vriK/uIr/mzm2+ebGkyT3MyPcmtLj\nFSlbyEmL58lZ4/jxS6u4991N/HXWWNffuNwI+HOX7KKwzM5fZ41rv65WTDI01EC9HSJP/gSglOL2\n7w3mxy+u4t21Ba5N06yrBLSM8EXXpJQiKyWOrJQ4pgw++fnFixeTO3EypVX1lNvrqW1oxF5vrido\ndPNNlxEWCxFWc30ixhHYYyKtxEeZ76OsFp/f29DYpKmw13Osqo6S6jqKK+sorqqjqKKWospajlSY\nr3X7SzhaUYu9vok3t2846TgJ0REkxUVii47AFh1BXHQEUVYLkVZFhNVCk9Y0OmZYVdc1mDec2gaO\nVdWRVbOduyPe4iyryZOvaBrOt5YLscSmEG1LZqTewSWlS7iy4T/U9BhFxeX/Ijmz/0mjyfrGJkqr\n6ykoqWZLYTnfHiznmz3H+N+FW/jfhVsY0SuRKydkcfm4LPd+SdXHTB577LXt7uK8OFvb0MQrN57W\n8UVM51z80n1tBnyAqUPSueu8oTz62XbG9k3mxskuVtd05vDjOr6p60BpDXOX7OKi0b04LaeDSpjO\nipn2Uohsezrq1MFpjOmbzN8X7+SK3KzO3/ydZRVkHr4IR0opMw00BO9ItloUyXFRJMdFdbqv1ppP\nFuUxdOypFFXUUlxVR3FlLSXV9ZRU11FaXU9lrQnmZTX11DumyDY0aSzKnMtqsRAfZSU5LorRtnKu\niHiJMU2fY49MYffQ29GjZzK871DOiG31u6qvgS0fEPvRXcTO+z5c8yb0Gn3CLpFWC+mOqb7j+qU0\nP55fVMWirYdZsOEgv/9wC498so3cDAvJA0sZ27ftgHuCgm/Mtu/pbf5O/rF0D498uo3sHnE8e30u\ngzI6GbnDLsM6AAAgAElEQVQmOdInZftP6kNLPztrIOv3l/Lwx1sZ0zep3UkPJ6g6YkblER3/Ped8\nbD5x3vf9Tlbtcr4h1ZS0e/+BUoo7vjeIG19ezfvrDrSfHnJyFk6TEb4QoUspRVykYmC6jYHpXizE\n3VgPy5+EJX82P0++k5jJv2RAR7M2ImNhzCzoeQq8dhW8OAOufBmGnNfp6bLT4vnplAH8dMoANh8o\nY943+3jnm31c+vdljO+XzA2Tcpgxsmf7+ed9K0BZoU/uiQ8XV/O/C7ewaOthLhjVk0evHIMt2oVw\nkuy4uNnGhduWLBbF41eN4cInl3L7G+v5+I4pJLV+I2yt6min0x2X7jjKRxsL+dU5Qzqf/tlcE7/j\nEsnTh2Ywqk8iTy/eyeXj+nQ8jbm5cFqYztJRSs0A/gpYgX9orR/x9zm7jboqs4ZndEJQCjH5RU0p\nFG4weeOiHWY5uNoK08+YZFODxZYJGSOg1xhIHQiWLnKfwP5v4MM7zKpRw38AMx6BJDdSLJkj4aZ/\nw+sz4a0fwew8s1asi0b1SeKhPqcwyVZEYUw2r3ydz+1vrKNHfBSXj+/DzFP7njxC37/KjMSjzDWA\nsup6nlq8g1eW78VqUfz2wuH8ZHKO6+m2uFSIjIPS/Z3umhgTyZOzxnHl3K+5//1N/O3qcR2fx1k4\nrR21DY387oNvyUmL5+azBnTeVmdNfHvHAV8pxe1nD2b2q2tYsOEgl4/v4G9qD+MRvlLKCvwdOBco\nAL5RSi3QWm/x53nDkr0cdufBzkVweDOU7IXqFossR8RCUh/oPwmyp8CAs4J2c4fbKo/At/Nh8zuw\nf+XxxxN6mVFWlA0ioqGswLwRVBdBk+MiZZQN+p8JA78HA8+GtMH+q8ToqZpS+Pf/wuoXIbE3zHoD\nhn3fs2Ml9IRr3oK5k+Cd/4KbvjSfANwQG6G4cXIOPz4zm/98d5Q3v9nPS8vyeX7pHvqlxnF6jlkp\nLj3ewpSC1RwePIsP8nbx1c6jfJNfQn1jE1eMz+K/zxtKzyQ3a8Qr5Ziaudel3cf1S+FX5w7h0c+2\nM3VIescpk6qjHS6u/tyS3ewpquKfN55GTKQLs7BarnrViXNHZDK8VyJPfbmTS8b2af9CcHMt/PDM\n4Z8G7HSsbYtSah5wCSAB3xWN9bD9Y1jzCuxZYoJcdKK5AWbYhZDS3xR4qq0wI+GinSZwrn0FLBFm\nFHnazdDvjNALggCFG2Hp47B1gVk6L2MkTPsNZE0wfWxvabnGelNXvXAjHFhj3gh3fG6e6zEIRlwC\nwy82nwCC2W+tYfO78Ol95k3qjJ/B9N94P7pLyITL5sK/fgif3Q8XPeHRYawW1Vwi5GhFLR9tPMjy\nXcV8sfUwb68pYLTaxbRoOw9tTOTj9dsY1jOB68/ozw/HZzGitxcBq9XNV5255ayBfLWjiAc++Jbx\n/ZLbv05QdRTip7b51L7iap5avJMLR/di6hAXlz9szuF3vuqVM5d/y7/W8uGGg+3PUgpyDl9prTvf\ny9ODK3UFMENr/VPHz9cDp2utb22xz2xgNkBmZmbuvHnzPD5fZWUlNpsX+dUQUVtSyNDSxfQq/Jzo\nuhLs0ekcyZhCcY8JlCcORVs6eJ/Wjdgq95B5eCk9D31BZEMVFbaB7Br4Y0pT/L6e/Ena+pvYKnaS\ns+d1ehxbQ4M1joO9Z3A4cxpVNjduXmklpuYwqcfWkH70a5JLN6Noojq2F0cypnAkYyrV8Z1cTHOB\nO/++Esu2MnDXSySVb6c8YRDfDfk5lQkDvW5DSwN2vUS//fPZPPLXFKWf6fLrOutHk9YcqtLkFH7I\nxAMv8s9hz5OYkkZytG9SZ4O/m0vGka9YNvlfLr+mxN7EA8triI1Q/M8ZsdiizBu5sy+qqYGz/vND\n9mRfzd7sWSf15/HVdnaVNvHHKbGkxLjYD93EWUsuZ2//K8nPaX+WUsvz/G5ZDQ0a5kyOPWGKsFPW\n/vkM2vUSSye/QWPEiVNlvYlf06dPX6O1ntDpjlprv30BV2Dy9s6frweeam//3Nxc7Y3Fixd79fqg\nKy3Q+pN7dcPv07V+IEnrf12p9fZPtW5s8Ox4tZVaf/OC1k+M0vqBRK1fu0rrI9t92+ZOnPA3KS/U\n+v2fmbY8kq31kj9rXV3i+5NWFmm9+mWtX/6B1g8mm/M9fabWSx7VuniXx4d16d/XgXVaz7vWnPPR\nIVqvfsnzv19n6mu1fnaa1o/017r6mMsvc/n/yZs/Mv92fG3pX8zvx17u1stW5xfrwb/5WF/z/Ne6\nrqFRa92iL+WF5pirnj/pdXM+3qL7/3qhfn3lXvfb+sd+Wn90l8u7L9xwUPf/9UI9f11B2zv8+yHz\nf7ux8aSnvIlfwGrtQkz299WuA0DLoVWW4zHRUvlB+OgueHIsrHyWo+lnwi9WwrVvwZDzweLhXZ9R\n8TDhRrj1GzjnQchfBs9MhEUPQl21DzvQiYY6+Oov8Ldc2PQ2TLoD7tgAU+9udy62V+J7QO6P4ccL\n4M6t5uJoZBx8+Qd4chzMnQyL/wgH15u0i7ca62HrQnjp+/DcWbBrsUlN3b4Wcm/w/O/XmYgouPhJ\nk3L4z2O+PbbW5npK39N8e1wwd9uCSxduW8rtn8qcy09h2c5i/rCwVVa4nZuu5q87wLNLdnPdGf08\nq1/fQXmFtlwwqidDMm387cudNDa18W/LWRo5SBMN/J3D/wYYrJTKwQT6WcA1fj5n11F+EL76P1jz\nMuhGc3PLlDvZtiGfnulDfXeeyBiY/CsYex0sesAE32/fhwsfh0Hn+O48bUgq/Rae/bXJuQ/9Ppz/\nMKS6MEPCVxJ6mtz5GT8zUwG/nW+uiyz5Eyx5xFRWzJ5kLnb3PQ3ShnR+EVRrqCg0AXHbx7DjM1Op\nMakfnPcwjL8+cMWxep4C466Flc/CqT/x3e+2bL/pY78zfHO8llpOzcwc4dZLr8jNYsfhCp79z26i\nIyycHusIqpWOxctbBPyNBaX8+t2NnJ6TygM/GOlZWzspr9CaxaK47ezB3PbGOt5fd4ArclvN2Ali\nHR3wc8DXWjcopW4FPsNMy3xRa/2tP8/ZJZQdMEF37T/Nhdix18DUuyAl27FDvn/Oa0uHS582c7oX\n/spc9Bt5OZw/BxJ7+fZcFYdh0YOM2/C6CYRXz4OhF/j2HO5K7geTbjdfVUXw3WfmYnj+V+YNEEBZ\nzN8hJdtMy4tNNoti11YwomAP7HnUTKt0jvpiU2HYD2D4RTDoXLAG4daW6b+Fze+bT25X/dM3x9y/\nymz9McJvefOVB+6ZMYzqukaeX7qHJSkWRuXa6dlcKTODhsYmXl+1j8c+206aLZqnrx3vefmLdkok\nd+TCU3rx0rI9PPTRFqYNTT+xxn87AX/RlsMUVTV51kY3+P1fp9b6Y+Bjf5+nSziyFZY/BRvfBLQJ\n9JPvhFQXbx33lZypcMsyWPZ/sPQJ2PEFnP1bM0L0dj5/Qx2snGtuLGqws7ffD+l/3d9MeimUxKeZ\nkfG4a82IvSQfCtebhbqPbDXBqGSvmYPdUAdR8dgaLRCdZWYBZYw089P7TAhOkG8psZdJk+XNgb1f\nQ/+J3h9z3woz5TXDw5FxR2wZ5k3UxamZrVktij9cOooJ2Snc8/Z6LnxyKXN6buJ84IOddcxduYyt\nheWcObAHcy47hR7eLKoSm+L2G5PFovjTD0dz4ZNf8eCCb3nqmvHHn2xj8ZNdRyu5fd46BiXBrAs9\nb6or5E5bf2tsMFMGV79g5tBHxpm87pm3mWmVwRIZA9PuhVOuhI/vgk9/DSufgWn3mcfczTs31MKG\nN8wnl5J8GDIDzp/Dnk376R9qwb41pcybbidvvKvy8pg2bVpg2uSuM281qcHP74ef/tv76aj7V5q7\na/3xZqaUueHMzRx+a5eM7UNVwXbmF8Syv2AfdVi5Y/5ueifF8vS147lgVE/v6y+5mcN3GpyZwG1n\nD+LxL77jkrGHOddZ7ry24vj8fsBe38itr68jOsLCf430fziWgO8PWsORLbDpHVj/OlQeMneHnv1b\nmPCT9ueXB0OPgXDdeya98eVD8P7NZtSfewOMurzzNUxL8mHLB7BiLlQcNPPnv/8YDD7XsYN3/6mF\ni6Li4ez74YNfmGsUw7wYKtrLzM19U+7yXftaa6cuvrt62yy8dctE9PxX0Tsz+PRnU8nuEe/ajVWu\ncObwnVX+3HDzWQP5aFMhv52/idNyUk1piNryEwZ6cz7eytbCcl68YQKWQ95Xle2MBHxfqauGglWw\n60vY+qGpMKgsMPg8GP8jsw3V8gdKwdAZpo1bF8BXT8Bn98Fnv4HsyaY0bkq2ycXXV0PlYfMxd1ce\nHN5kjtHvTLjkKXO3ayje5NUdjJ5lbmRbPAeGXOD5TJDvPjc3wvnzgn5yP3PjnI+oqiKULd13i6E7\nxSSbCRW1FW7fHRsVYeHPV4zmsqeXc/nTy/jrrHGMarGA+aebC/nn13v5yeQczh6WSZ4E/BBVWwnF\nO80o6NBmOLgWDqyFpnpzh2vOWXDm7WaU1VXKG4AJECMvNV9HvzOlDrYuhJXPQWNtq30jTP76vIdM\nPwM580a0zRoBZ90L7882b9wjL/XsOFs/MGUtsk71bftaSupr7j6uq26u0+OVqqMuL23olpYlkj0o\nhzA6K5l/3nga//3WBi79+zK2xpTyXTE88MxyVu8tYXRWEr+e4Xo9JG9174CvtVmQoLbC8VUJdY7v\n7WXmq/qY+cdUVWRSFiV7oebY8WNExELPUTDxF2Y03Pf0oNXJ8Kn0IaYMwPTfmFV6Kg+Zj+BR8Y4a\nN6ldp2hZd3LKFbD0Mcj7oymt4e61mLoq2LEIxl3n37+vc6ZO+QFT/8hbVUfNlFpfa1kiOdmDefzA\npEFpfPrLKfzP++uJ3FHLZzurqUxv4O7zh3L1af1cWynLR8Iz4NfbTXAq3Wu+ygtNwKo8AtXFJojX\nlJh8mu5kKpSymhkd8elmtN5rrPnDpw6AzFHmQp+/bqwJFRaLKfqV2DvYLRGdsVjNxfh3bjRTTU+5\nwr3X71xkVnka/gP/tM/JWSW0rMD7gK+1Y4Tv2lq2bnGxRHJnkuOiePLSAfAoXDd9NHee03bNH38L\nj4Bfug9Wv8iorctgwy/NhURa3OWmrCZY2zLNP4rUAeYPGZNk8mnRCWaqVHSCmYoWnWDe2WOSICp4\nd8UJ4ZERl0GGY5Q/4lL3ZtpsWWA+vfWf5L/2wYkB31t1ldBg90/6NKbFCN9Lyl4GQEZ6ptfH8lR4\nBPzaClj+N2JjekP2eHNjUeoAc0dfcj/zDyHcR+FCOFksMP1+ePNaUzn11J+49rqGWjNba6SbbxKe\nSOwNKN8EfDfWsnVbyxy+txwBP2B3YbchPAJ++nC4/xDfLF0WuvOkhQikYReaUfriOSat40qQ2Z1n\nrmGNuMTvzcMaaab8+iTgO++y9UfA990IvzngRwfvGl945CosltCd8ihEMChl6hZVF5mpmq7YssAE\no5wA5ZeTsjwur3CC5hG+H3L4kXFgjfI6hw+ExAg/PAK+EOJkvcfBmKthxTOO61odaKyH7R+ZO6Qj\nvChF4I6krNBP6SjlUT2dNjWvdiUBXwjhD2f/j5m0sOjBjvdb8YwJaqdcGZBmASbglx/wvkR1pSPg\nx/lhhA8mjx8mOXwJ+EKEs6Q+prDat+/D+jfa3qdoByx+GIZe2KIkRiDa1tfMrqku9u44VUdNKirS\nzfV1XeVhPZ2T2MvM3fdRwVuVLzwu2goh2jflv2Hf17DgVlMiu+V/+6ZGU38nIsasjRvIshjNUzP3\ne5d/99ddtk6xKWbtCm/Zy8wbUxCnecsIX4hwFxEFM/9lZrO9+SNsFTuPP7fyWVMZ84I/dV4oz9cS\nHQt9e5vH93fAj0n23UXbIKZzQEb4QnQPMYlw3Tvwj3MZu/53cOAVM7ov3mEu1I6eGfg2NS+E4m3A\nLzJVX/3Flzn8IJdd8dsIXyn1oFLqgFJqvePr+/46lxDCBQk94fr3OZY61oyuU3NMoP/Bk8GpcBqX\nampRhfoIPzbZzLBpbPDuOPby43fuBom/R/h/0Vr7eHVlIYTH0gaxZeQ9ZITCDYrOhVC8mYuvG81F\nX3/n8MGM8r251mAvC/zqdq1IDl8IETxJWWaNZw9F1ZUC2r9lyJsLqHk5UycEcvj+Dvi3KaU2KqVe\nVEqldL67EKJb8fLmq+ja4uPH8ZdYxwp1YRDwvUrpKKUWAW1d2r8feAb4A6Zs5R+Ax4Eb2zjGbGA2\nQGZmJnl5eR63p7Ky0qvXh4pw6QdIX0JRKPWjf0kDOZWHWPLlF2iL++VRbI5PB6u/O0hlYZ6PW2ck\nlO8mF9i0cgnFu6o9OoZqauSsugr2HCphbzu/+4D8XbTWfv8CsoHNne2Xm5urvbF48WKvXh8qwqUf\nWktfQlFI9WPtq1o/kKh18W6PXv7dq/9tXl9xxMcNa6F4lznHutc9P0ZVsTnG10+3u4s3fxdgtXYh\nFvtzlk6vFj9eBmz217mEEF2Ul3Xxo2uLTXGzuB4+bFQrzTn8Yx3v1xFnHZ0gVsoE/87S+bNSaiwm\npZMP3OzHcwkhuiIv5+JH1xaZJTf9efdqdJIpieBNDj8E6uiAHwO+1vp6fx1bCBEmnMtmehzwj0GS\nn5fetFjM/PlqL0b4IRLwZVqmECJ4ImPNHPpyL0b4gVhrOS41LEb4EvCFEMGV2MezEb7WJocfiIAf\nm+JdDl8CvhBC4Plc/JoSLLoeEgIR8L0d4Qd/8ROQgC+ECLakvibgu7sQSrnjDt1AjfCrvU3pqKDP\n0pGAL4QIrqQsqKt0fwTtrFHvLLPsT77I4UcnBLUWPkjAF0IEW0q22Xa27m5rAR3hp0JdBTTUefb6\nECirABLwhRDB5qwgWbLHvdeVF6KxgC3T921qLdZR1tjTuvgS8IUQguMj/GO73Xtd+UHqopLBGoB1\nnOIcBdQ8nYsvAV8IIYCoeHO37DF3R/gHqI32Y0mFlrwtkVwrAV8IIYzUAe6P8CsKqY32YkESdzSX\nSJYRvhBCeCc1x6OUTm10qn/a05q3I3wJ+EII4ZCSA5WHoa7Ktf3t5VBbHrgRvjc5/KYm094gz8EH\nCfhCiFCQOsBsXc3jVxQCBC6HH2UDS4RnI/y6CkDLCF8IIYAWAd/FtI7jpquAjfCVcpRX8GCEHyJ1\ndEACvhAiFDjn4rsd8AOUwwdHATUPRvghUkcHJOALIUJBTBLEpbkd8OuiApTSAZPH9ySHLyN8IYRo\nJTXH9bttKw5CXA+arFH+bVNLsSlQ48GdtuES8JVSVyqlvlVKNSmlJrR67j6l1E6l1Hal1PneNVMI\nEfZSB7h+0bb8YGBq6LQkOXw2A5cD/2n5oFJqBDALGAnMAJ5WSlm9PJcQIpylDjBlkuvtne9bfiAw\ndfBbik32MIcfJgFfa71Va729jacuAeZprWu11nuAncBp3pxLCBHmUgcAGkr3dr5vMEb4calQX+3a\nG1JLtY6LtiEwD99fVYf6ACta/FzgeOwkSqnZwGyAzMxM8vLyPD5pZWWlV68PFeHSD5C+hKJQ7UdC\neQm5wKYlH1Cc1v740NJYx9TqYvYU11GpAteXXgePMhRY/uVH1Lkx/3/gzs30ssbw1dKvOtwvEH+X\nTgO+UmoR0LONp+7XWn/gbQO01s8BzwFMmDBBT5s2zeNj5eXl4c3rQ0W49AOkL6EoZPtRdQqsvYdT\n+thg4rT29zv6HSyFnLFT2FtqC1xfvi2F757hzLHDIHOk668rexvKe3TazkD8XToN+Frrczw47gGg\nb4ufsxyPCSFE2+JSITqp86mZhzaabc9RUFrs/3Y5eVpPJ0Tq6ID/pmUuAGYppaKVUjnAYGCVn84l\nhAgHSrlWRK1wA1ijIH1YYNrl5Gk9nXAJ+Eqpy5RSBcBE4COl1GcAWutvgbeALcCnwC+01o3eNlYI\nEeZcKZN8aCNkDAdrZGDa5OTNCD8ELtiC97N03tdaZ2mto7XWmVrr81s897DWeqDWeqjW+hPvmyqE\nCHupA6B0HzTWt/281lC4EXqODmy7wPOa+Pby8BjhCyGET6UOAN3Y/oLm5QdMwO01JqDNAiAyFqzR\nksMXQgif6D3ObPetaPv5QucF2yCM8JVyv56O1hLwhRCiTRnDTRG1/KVtP39oI6DcmxbpS+5WzKyr\nMp9YYsIghy+EED6lFGRPhj1Lzei4tcKN0GMQRNsC3zZw1NNxI+BXF5ltXIDq9ndCAr4QIrTkTDHV\nMNuarXNoI/QKQjrHyd16OpVHzTY+3T/tcZMEfCFEaMmearat0zrVx6Bsf3Dy907u5vCrjpitTQK+\nEEKcLG0w2DJNWqcl5x22QR3hO3L4baWb2lLpCPjxGf5rkxsk4AshQotSkD3FjPBbBtbmGTpBmJLp\nFJsKjbWmaqYrqhw5fEnpCCFEO3KmQOVhKNpx/LFDGyGxD8QHcFnD1twtr1B1BGKSISKAK3N1QAK+\nECL0ZE8x2/wWaysF6w7blpypGWduvjOVR8AWGukckIAvhAhFqQPMaN6Zx6+tgOIdwc3fAyQ4KsVX\nHHJt/6qjIZPOAQn4QohQ1JzH/wq+/js8dSroJjNHP5gSepltRaFr+1cekYAvhBCdypliblz67Dfm\nZqsffQA5U4Pbpvg0UFY3RvihldLx1xKHQgjhnRGXQPFOGPp96BsiS2JbrCaAuzLCb6g1dXRCZEom\nSMAXQoSq6AQ458Fgt+JkCT1dG+E7p2SGyE1XICkdIYRwT0IvFwN+aN10Bd6veHWlUupbpVSTUmpC\ni8ezlVI1Sqn1jq+53jdVCCFCQEJP11I6IVZHB7xP6WwGLgeebeO5XVrrsV4eXwghQktCL6guhoa6\njm+oCrE6OuD9EodbtdbbfdUYIYQIec65+JWHO94vxOrogH9z+DmOdM4SpdQUP55HCCECp3kufid5\n/KqjEGWDqDj/t8lFSndS9U0ptQjo2cZT92utP3DskwfcpbVe7fg5GrBprYuVUrnAfGCk1rq8jePP\nBmYDZGZm5s6bN8/jzlRWVmKzBWlhBB8Kl36A9CUUhUs/IDh9sVXsZsKaX7F55L0UpU9sd7/hWx4n\nsfw7Vp7RVsb7ZN70Zfr06Wu01hM63VFr7fUXkAdM8PR551dubq72xuLFi716fagIl35oLX0JReHS\nD62D1JeKI1o/kKj1imc73u/lH2j9j3NdPqw3fQFWaxditV9SOkqpdKWU1fH9AGAw0MbyNUII0cXE\n9QBLROczdUKsjg54Py3zMqVUATAR+Egp9ZnjqanARqXUeuAd4BattRvLxAghRIiyWMDmws1XIVZH\nB7yclqm1fh94v43H3wXe9ebYQggRsjqbi9/YYKZuhlAdHZA7bYUQwn0JPTuellldDOiQG+FLwBdC\nCHcl9Op4hN9805WM8IUQomtL6GkWM6+3t/18CN50BRLwhRDCfc1327Zz4TbEFi93koAvhBDu6myp\nwxCsowMS8IUQwn2dLXVYeQSs0RCdGLg2uUACvhBCuKuzejpVR80FW6UC1yYXSMAXQgh3xaaANarj\nEX6I5e9BAr4QQrhPqY6XOgyxxcudJOALIYQnOlrqsPIoxKcFtj0ukIAvhBCeaG+E39QE1UUhNwcf\nJOALIYRn2iugZi+FpgZJ6QghRNhI6Am1ZVBXdeLjxxyV4BP7BL5NnZCAL4QQnmhvaubeZWbb9/TA\ntscFEvCFEMITqQPM9vDmEx/fuxx6DIKEzMC3qRMS8IUQwhN9ciE6CXZ8cfyxpkbY+zX0nxS8dnVA\nAr4QQnjCGgEDp8PORWDW7obD35q8fjgGfKXUo0qpbUqpjUqp95VSyS2eu08ptVMptV0pdb73TRVC\niBAz+Fxzt60zrbN3udlmh2HAB74ARmmtRwPfAfcBKKVGALOAkcAM4GnnouZCCBE2Bp1jtjs+N9u9\nX0FyP0jKCl6bOuBVwNdaf661bnD8uAJw9vISYJ7WulZrvQfYCZzmzbmEECLkJPSEnqNhhyOts3c5\n9J8c7Fa1y6tFzFu5EXjT8X0fzBuAU4HjsZMopWYDswEyMzPJy8vzuAGVlZVevT5UhEs/QPoSisKl\nHxAafcmJGkK/fe+xbsFcxlcXs83eg0MetCkQfek04CulFgE923jqfq31B4597gcagNfcbYDW+jng\nOYAJEyboadOmuXuIZnl5eXjz+lARLv0A6UsoCpd+QIj0ZUAMvPgO40s+AmDY+TcyzDll0w2B6Eun\nAV9rfU5HzyulbgAuAr6ntfNSNQeAvi12y3I8JoQQ4aXPBIhJgvyl5maslJxgt6hd3s7SmQHcA1ys\nta5u8dQCYJZSKloplQMMBlZ5cy4hhAhJ1ggY+D3zff9JIbfoSUveztJ5CkgAvlBKrVdKzQXQWn8L\nvAVsAT4FfqG1bvTyXEIIEZoGn2u2/c8Mbjs64dVFW631oA6eexh42JvjCyFElzD8Yji0GUZeFuyW\ndMiXs3SEEKJ7irbBjDnBbkWnpLSCEEJ0ExLwhRCim5CAL4QQ3YQEfCGE6CYk4AshRDchAV8IIboJ\nCfhCCNFNSMAXQohuQh2vdxZ8SqmjwF4vDpEGFPmoOcEULv0A6UsoCpd+gPTFqb/WOr2znUIq4HtL\nKbVaaz0h2O3wVrj0A6QvoShc+gHSF3dJSkcIIboJCfhCCNFNhFvAfy7YDfCRcOkHSF9CUbj0A6Qv\nbgmrHL4QQoj2hdsIXwghRDu6fMBXSr2olDqilNoc7LZ4SynVVym1WCm1RSn1rVLqjmC3yVNKqRil\n1Cql1AZHX34f7DZ5QyllVUqtU0otDHZbvKGUyldKbXKsULc62O3xhlIqWSn1jlJqm1Jqq1JqYrDb\n5Aml1FDH38P5Va6U+qVfztXVUzpKqalAJfBPrfWoYLfHG0qpXkAvrfVapVQCsAa4VGu9JchNc5tS\nSgHxWutKpVQk8BVwh9Z6RZCb5hGl1J3ABCBRa31RsNvjKaVUPjBBa93l564rpV4Blmqt/6GUigLi\ntCaVF1UAAAJBSURBVNalwW6XN5RSVuAAcLrW2pt7ktrU5Uf4Wuv/AMeC3Q5f0FoXaq3XOr6vALYC\nfYLbKs9oo9LxY6Tjq0uOLpRSWcCFwD+C3RZhKKWSgKnACwBa67quHuwdvgfs8kewhzAI+OFKKZUN\njANWBrclnnOkQdYDR4AvtNZdtS//B9wDNAW7IT6ggUVKqTVKqdnBbowXcoCjwEuOVNs/lFLxwW6U\nD8wC3vDXwSXghyCllA14F/il1ro82O3xlNa6UWs9FsgCTlNKdbmUm1LqIuCI1npNsNviI5Mdf5ML\ngF84UqJdUQQwHnhGaz0OqALuDW6TvONIS10MvO2vc0jADzGOfPe7wGta6/eC3R5fcHzUXgzMCHZb\nPDAJuNiR+54HnK2U+ldwm+Q5rfUBx/YI8D5wWnBb5LECoKDFp8Z3MG8AXdkFwFqt9WF/nUACfghx\nXOh8AdiqtX4i2O3xhlIqXSmV7Pg+FjgX2BbcVrlPa32f1jpLa52N+bj9pdb6uiA3yyNKqXjHZAAc\n6Y/zgC45u01rfQjYr5Qa6njoe0CXm9zQytX4MZ0D5mNRl6aUegOYBqQppQqAB7TWLwS3VR6bBFwP\nbHLkvgF+o7X+OIht8lQv4BXHrAML8JbWuktPaQwDmcD7ZlxBBPC61vrT4DbJK7cBrzlSIbuB/wpy\nezzmeAM+F7jZr+fp6tMyhRBCuEZSOkII0U1IwBdCiG5CAr4QQnQTEvCFEKKbkIAvhBDdhAR8IYTo\nJiTgCyFENyEBXwghuon/B/a5zgq1LoouAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8e84c29f10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting your derivative",
    "\n",
    "scalar_space = np.linspace(1, 7, 100)",
    "\n",
    "\n",
    "y = [s.run(weird_psychotic_function, {my_scalar: x, my_vector: [1, 2, 3]})",
    "\n",
    "     for x in scalar_space]",
    "\n",
    "\n",
    "plt.plot(scalar_space, y, label='function')",
    "\n",
    "\n",
    "y_der_by_scalar = [s.run(der_by_scalar, {my_scalar: x, my_vector: [1, 2, 3]})",
    "\n",
    "                   for x in scalar_space]",
    "\n",
    "\n",
    "plt.plot(scalar_space, y_der_by_scalar, label='derivative')",
    "\n",
    "plt.grid()",
    "\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Almost done - optimizers\n",
    "\n",
    "While you can perform gradient descent by hand with automatic grads from above, tensorflow also has some optimization methods implemented for you. Recall mommentum & rmsprop?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_guess = tf.Variable(np.zeros(2, dtype='float32'))",
    "\n",
    "y_true = tf.range(1, 3, dtype='float32')",
    "\n",
    "\n",
    "loss = tf.reduce_mean((y_guess - y_true + tf.random_normal([2]))**2)",
    "\n",
    "\n",
    "#optimizer = tf.train.MomentumOptimizer(0.01,0.9).minimize(loss,var_list=y_guess)",
    "\n",
    "\n",
    "# same, but more detailed:",
    "\n",
    "#updates = [[tf.gradients(loss,y_guess)[0], y_guess]]",
    "\n",
    "#optimizer = tf.train.MomentumOptimizer(0.01,0.9).apply_gradients(updates)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt41OWd///neyYHDuEczpBwBjkIQjio3Vbb2iratVqr\noLWr1sXt2v6233Z3e3Dr9rBuu9tuW7vtqnio9YT1XE9Ue7CCSoAM5/MhMEkgkJBMICGBZGbu3x8z\njEkIZAJJZjJ5Pa4rlzOf+WTmnci85s5935/7NuccIiKSWjyJLkBERNqfwl1EJAUp3EVEUpDCXUQk\nBSncRURSkMJdRCQFKdxFRFKQwl1EJAUp3EVEUlBaol44OzvbjRkzJlEvLyLSJfl8viPOucGtnddq\nuJvZaOAJYCjggKXOufubnXMZ8HtgX/TQS865H5zteceMGUNBQUFrLy8iIo2YmT+e8+JpuQeBbzjn\n1plZH8BnZn90zm1rdt5K59w1bS1URETaX6t97s65UufcuujtamA7MLKjCxMRkXPXpgFVMxsDXASs\nbuHhS8xsk5ktN7Np7VCbiIico7gHVM0sC3gR+Jpz7lizh9cBOc65GjNbCLwCTGzhOZYASwBycnLO\nuWgRETm7uFruZpZOJNifds691Pxx59wx51xN9PabQLqZZbdw3lLnXJ5zLm/w4FYHe0VE5By1Gu5m\nZsCjwHbn3M/OcM6w6HmY2bzo81a0Z6EiIhK/eLplLgVuBTab2Ybose8AOQDOuQeBG4Avm1kQqAMW\nOW3xJCLtwOcPkF9YwYJxg5iTOyDR5XQZrYa7c+49wFo551fAr9qrKBERiAT74qWraAg5vB7j7svH\nM31kfw5W1VESqOXKacOYM2ZgostMSgm7QlVEpDVLV+ylPhTpBAiGHff/eU+Txx9ZuY+FM4Zx8/xc\n0jxGgT+gFn6Uwl1EkpLPH+CtrYdj99O9xi9umsXqwkqeWu0n7CKXzP9h62He2Hwodp7XjBvzRrFg\n/CBGD+zF6AG9yM7KYF1RVbfq3lG4i0hSyi9sOidj6vC+DOvXk2svGslzvmIagmHS0zw89ndzed5X\nwsvrDwAQco5la4tZtrY49r0ZXg8NoTAO8BhcOX0YU4f3ZVBWJoN6Z3Ckpp6SQC2fmDIk1s3T1fv6\nLVHjnnl5eU5ry4jImfj8AW55JJ8TDeHYsTSP8YNrpzN5WJ8mwXvq3FOB/5vb5jK4TyZFlbUUVdTy\n2qZSfP5A7HkyvB7qQ+GWXpZx2b0Z0b8n+YUVhMKOzDQPT//9gqQJeDPzOefyWj1P4S4iycrnD/CL\nP+5i5Z4jsWMegx9eO52quoYmreqztbSbh//Tdy5g2oi+VB6v58G/7uXJfD+OyMyR3OzeBI6f5Ghd\nMPb9M0f14+b5ORypOcmCcdkJDXqFu4ikBJ8/wE0PrSIYbppVBm1qVZ8p/FsKfoBbHs6nPhTGMMwc\nwWhDP81j/OzGmfztrMQssaVwF5GU8czqIu79/RbCYQcGjXN+8tAssrMymTGyH316pp9TH3lLwd/4\n2Lu7yvjfP++hcVpOHJLFVdOHcdWM4dSeDJK/r7JT+ucV7iKSUk6F7YBeGfzg9a3UB8O46IyZxnqk\nR1rf7RmyjVv3aV4Pt16cy5YDR1mzr5Lo5w0A6Wkelt05v0Pn3scb7potIyJdwpzcAbHAHjOoF499\nsJ8VO8ti8+BPaQiGyS+saNdwn5M7gKfvXHBa6768+iT3vLyZt7dFpmzWB8Pc8du1LJqbw5XThxEO\nu05r0TencBeRLsHnD/D21kMcOnaCd3aUcexEkJH9e3Cg6kTsHCPSel4wblC7v37jD5dTBvfJ5K6P\njWfF7nIagmE8HmPc4Cwee38fD60ojNWU2QF/TbRG4S4iSe/P2w+z5EkfoWhn+yXjBvH1T01iTu4A\nlq0pZvmWUqYN73vOfe7no6VW/dG6Br790ibe3HwIB5xsCPPiuhKFu4iIzx/gvd3llFWf5PmCkliw\newwunZhNXrRf++b5Odw8P7H7QzRv1ffrmc6XPjKOP247TEPI4YDn1hbzudmjOi3g27QTk4hIZ/D5\nA9z8cD4//9Nunl5dxLjBvclM8+A1yOigbpf2Nid3ADfmjY4NtgbDjvv/tJvOmsSilruIJJ38wgrq\ngx9eQTondwD3XTcjqZYDiGd5gutnj+LFdSU0BCNLH6zYXc4tD69m3riB/M3EwR36cyjcRSTpDOiV\ngRmcauQ+X1DM9bNHcfflExJbGLC6sIJH39vHn7cfJuQiFzX93cW5zM4dSOB4PUWVtVwxbSgeM/IL\nK7j3mmkEautZMHYgL68/yFOr/XxQWMH/vbOHZUsu7rCAV7iLSFJ5d2cZ3/39liYXKoXCrt2nN7aV\nzx/gyVX7eWXDwSbHg2HHo+/v59H398eOLV1ZGLud7jW+9omJ/K6gmNX7KmPH60OuQwdZFe4iknA+\nf4AP9h4hcLyeZ9YUxQZPoWOnN7alvpsfzudksOliY6emOT5wy2ze3XWEJ1btp9kqCTSEHD95e1eL\nz3vWXZDOk8JdRBKqYH8lix/OpyF6MdK47F4cqDpBMBTG6zE+nzea6ztxlklLfre2qEmwn/rAuWHO\nqNgMmL49M3h2bRENwTBerweci3xImTWZ6WNmhMOO9DQP188e1WE1K9xFJCEOVtXxfEEJj75XGAt2\nj8Hn5oxiwbjspBk8fXdnGS/4SmL307zGjXmjT5vW2Hy+O9BkuYRTC5PF+uA7+GdTuItIhzs1s2Tu\nmAFUHq/n2bXFvLurHOfgwpH92H7oWKw1e2pJ3Y4Mvnhmuvj2V/LIe/v4y/ayWFeLATfljea+62a0\n+D3N6z51u/n6851B4S4iHepUf3V9dDogwLC+Pfjq5RP4fN5oRg/s1aG7HjV+7otG9+cPWw7xT79b\nTzDkSPd6+NoVEwmFHZeMzwZg5e5yigO1vLTuAC66KFia13Dn0ZXS0R9WLdGqkCLSIQrLa3h9UylP\nrNrPkZr62PHPzR7Jf98wE6+nI4cTI3z+AIujHyxm0CPNS11DqE3P4TVYNC+HEf17JkU3kVaFFJFO\n5fMH+MOWUo7Xh1hfVMX20mOYwZShfaiqbcC5SMv35vm5nRPs+yv5lxc3xS6Gcg4uGN6HeWMH8tj7\n+wlGN+IItdDANcDrsVjNiR7QPRcKdxE5JyeDIbaXVrOhKMBfdpSxcveRWLfLpKFZ3HvNVBbOGM6w\nfj06fbPpt7Ye4stP+WJ95R4gI93DPVdPZU7uAK6YOuy0wc7GM1w6c+CzoyjcRaRFjQN5dk5/iipr\n2VBcxfqiKjYUV7Ht4LHYJtNZmd5YsHsMrp01kjs+Mjb2XJ3V57x2fyUP/HUv7+76cBDUY3DphGy+\n9slJsRoa19N4sBNImlk650vhLiJNhMOuyaCjGfTOTKP6RGTD6J7pXmaM6sftl45h1uj+zMrpz8Gq\nE032Ie3oC44af/DMGNmPjSVVvFBQwnMFxZGNri1yZeipGTiNg725M81w6eoU7iLdlHOOsuqT7DxU\nza7D1bH/7jpc02TQ0TnIHdiLxfNzuGj0ACYNzSLN23RB2eH9era4U1FHaDz7JhLintOuHPUAN+aN\nTppB0ERQuIt0Ayt2lrN86yGyeqRRVx9k16Eadh6u5mhdQ+yc7KxMJg/LYtG80fRI8/Do+/sJhSIt\n8e9fO73VgOysrpc/bT8cC3PnYOqIvtz10fH0TPdw11O+2F8PXXEQtD0p3EVSiHOOkkAd20qPsfXg\nMbYdPMaG4kCTqYi9MrxMHd6Xqy8czuShfZg0tA+ThmYxKCuzyXN9MjromEwtX58/wGsbIwt3eaJr\nu/9bdJAU6LS/HroChbtIkmhtRknzxxtCYXYfrmFbaSTEtx48yrbSY7G+cY/BuMFZDM7KpKKmHhc9\ndvfl47n78omt1pOIC2/OxucPsGjpKhpCDq/HuGluy0sAJFPNiaRwF0kCq/Ye4dZH1xAKuxY3U15d\nWMGtj62JbMJsRs6gnhwInIjNVumR7mHKsL58ZuYIpo3oy9ThfZkyrC89M7z4/IFmg53Zifoxz0t+\nYUVsDRqcY2T/ngrys1C4iyRQbX2QZ1YX8fM/7iIYnbvXEAzz3u5yGkJhVhdWsmZ/BasLK2OPh5zD\nObj9I2OYOrwv00b0Y2x27zNeGNTSBs5dUVamN3Y70UsAdwWtLj9gZqOBJ4ChgAOWOufub3aOAfcD\nC4Fa4Dbn3LqzPa+WH5Du7P09R3jw3b1sKK6i+kSQCUOy2H/keHSJWPCaEQxHpiFOGdaXsdm9+OO2\nw7Gpfc1b9qnO5w+weOkq6kMOrxk//Oz0hG+KnSjtufxAEPiGc26dmfUBfGb2R+fctkbnXAVMjH7N\nBx6I/ldEok71mQ/omc49v99C43bVvmiwm8H4wVl8fMoQ5o0ZyNwxA+nXK73J9yd96/vpp+Gee6Co\nCHJy4L774JZbzuspm3TJ4AjU1p/1fIkj3J1zpUBp9Ha1mW0HRgKNw/1a4AkX+TMg38z6m9nw6PeK\ndHvv7DjMkid9jQLqQyP79+C6i0Yxb+xAZucOICuz5bdllxgsfPppWLIEamsj9/3+yH04r4BfMG4Q\n6V4P9dENPNQl07o29bmb2RjgImB1s4dGAsWN7pdEjyncpVsqO3aC1fsqWb0v0l++u6zmtHM8Bhle\nD79cPDv5Qzte99zzYbCfUlsbOX4e4T4ndwCP3z6XLzy6mqtmDE+d31cHijvczSwLeBH4mnPu2Lm8\nmJktAZYA5OR0z/4ySU0Hq+piQb56XyX7jhwHoHeGl7wxA5k3diDPri0iOrkFrwcWzc1JvQttiora\ndrwNLpmQzeShfVi9rxKfP5Bav7cOEFe4m1k6kWB/2jn3UgunHABGN7o/KnqsCefcUmApRAZU21yt\nSIJF+r2PMHZQFjX1QdZEW+fFlXUA9OmRxrwxA1k8bzTzxw5i2oi+bCw5yvItpaR7PYTC0XR3MCIV\np/Ll5ES6Ylo6fp58/gC7y2oIhh03P5zPM3/fvQaV26rVcI/OhHkU2O6c+9kZTnsV+IqZPUtkIPWo\n+tsllTjneMFXwrde2hzb7BhgQK905o0dyO2XjGX+uIFMGda3yZTExptFQNPFrFKy3/i++5r2uQP0\n6hU5fp7yCysIR0ehG0Jh8gsrFO5nEU/L/VLgVmCzmW2IHvsOkAPgnHsQeJPINMg9RKZC3t7+pYp0\nviM1J3ll/QFe8JWw41B17LgBt106hu9ePRVPC/PLT81seX9PeSzYPQafzxvNyFRezOpUv3o7z5aB\nyKBqRpqHEw1hnIMBvTLO+zlTmbbZE2mmIRTmLzsiO96/s6OMYNgxc1Q/5o0bxBMfRHbwOdtcc58/\nwC0P53MiGupGJNi74/z09vbM6iLueXkzjshg9LIl3e/3qW32RNpoe+kxXvCV8Mr6A1Qcryc7K5M7\nPjKWG+aMYtLQPgBcOa3lxbR8/gDv7S7H6zF+v+Fgk2BfNG80owb0St3WeicK1NZjFlkNsj4U5r/+\nsJ2nvrSAjDRP69/czSjcpdvy+QO8s6OMuvogq/dXsuXAMdK9xicvGMoNc0bxsUmDT1u3vKW55j5/\ngBsfWhXri++d4Y3tv5mR5uGGOaMV6u3kVNdMQzAMZqzZF+Czv36fuz46jpKqOn2ANqJwl27pnZ1l\n3Pnbglggj83uxfc+M5W/nTWSgb3b1pf70rqSJoOsfztrBDfMGc1L60rQlLD21XydnIqak3zj+Y38\n0+82YNDiomvdlcJdupUTDSEee38fv/jj7lggewxumDOa2y4d28p3t6x5gL+y/iAHq+r4YG8FobDj\nhYISbsgbddrytI2XE4C2793ZZZYjOE/Nf87mfz3dWlzF//11Lw6oD2oWzSkKd+kWQmHHy+sP8D9v\n76T06Anycgew+cDR2ODo+UxL/NzsUbxQUExDyOHxwEU5/VmzLxBbaqA+FOaZ1UUsW1PEtOF9Gdwn\nk9r6EGv3VxJ2RKdOOsLhyJZx37l6ClOH92NPWTVbDx5j6vC+TBzaB+ccYQcOx65D1fznmzsIhsOk\nez08c+d85owZ2E6/rc7l8wdYtfcI00b0ZfTA3lTV1hOobSBwvJ7NB47yzJoiQmFHmsf40kfGkjdm\nIEP6ZDKkbybZWZl84oKhPLyyMPL7dqTmFNNzoNkykvJW7i7nP9/cwfbSY1w4qh/fvuoCLh4/qF1b\nvs2fq2B/Jbc8svq0vT1H9u/BoKxMDlbVNdkd6Xz1SPcwc1R/pgzrw+RhfZk8rA+Th/Vh56HqhLTu\nm/8+wmFHec1J/BW1FFXWUlRxHH9lLdtLj7Hr8OlLM7RFnx5p1JwIxv6Cmjq8D9fMHMFlk4ZQcfwk\nm0qOxupo6f+5zx/gxXUlGHSJK4bjnS2jcJeUte3gMX60fDsrdx9h1ICe/OuVU7hmxvAW56V3BJ8/\nwEvrSni+oJhQs6V6G2+g4fV4cLho69TDPVdfwNaDR3nBV0LYneo2GsVnZ40EA48Zuw9X88PXtxMM\nRzbvuHxyJMh2HqrmeP2Hm1tH/iaANI/x/31iIgtnDGPMoN5sLDnapi6hs30Q1gfDVB6v50jNSSqP\n1+PzB/j1O3tiq1yO6N+T8uqTTT7oPNHjHjOKKmtjtV594XA+nzeagb0y6N8rHX9lLXf+dm1so5Ff\nLrqIoX17UF59krLqk5RVn+CdHWVsLDl61v8XZpHQ33GohnA4spPT312cy+FjJ3h986HYeRlpHpYl\n+ZWvCnfptkqP1vHTt3bx0voS+vZI56sfn8CtF+eSmeZt/Zs7wJmC8Wx97s13T2ppkLCl5w2HHQeq\n6thxqJonPtjPyj1HTqsnzWuEQi627Z5BrHvoxrxRDO/XE0dkumHYOQ4erePldQcIhR0eMy7K6U/Y\nOSqP11NRU0/1yeBZf/4JQ7K4fPJgcgb2ImdQb3IG9mJk/55kpHnO+eds/vgtj+RzsiHcZPzD67Em\nA90ZaZ7YBWVnYsA/f3oyd18+4aznJZLCXboVnz/Au7vKKKms443NpTjg9kvG8I+XTYith97VnG+3\nUZPg9Hr4j+umYxjPri1i7f7AOdc1OCuDScP6MLB3JoN6ZzCodwYDsyL/HZSVSenROv7l+U2tXuzV\nXj/nqec49VfSqR2rwucQbWq5twOFu7SXtfsrWbw0P/am/ujEbP7z+hmMGtArwZUl3pn6mGNdQl4P\nOBfrNvrt7fOYnTsAA8wMj8G6oqpWW9fxvG5nOPW6B6vqWLamKBbww/pkcrj6ZKxlv2DcQD4+ZQg/\nfXsXwWAYM/jEBUO562PjkzrYQeEu3cS2g8f40m/XUnr0BBDpZvjGp5L7z+pk0NZpmF1t2mXz7p57\nr5nGD17fetoHVFf7uUDhLimutj7I/X/azSPv7SMr00ttfajb7i8qLWse3F0xyFuitWUkZb2zs4zv\nvrKFkkAdi+aO5ltXTWFv+fGUeONK+2l+sVOX2KawHSncpcsoqz7BD17bxuubShk/uDe/W7KA+dFu\nhTm5Gd3qjSvSGoW7JL1w2LFsbRE/Xr6Dkw1hvn7FJO762LiETW0U6QoU7pK0fP4Ar288yAeFFew8\nVM3F4wZx33XTGTc4K9GliSQ9hbskpZW7y7ntN2tjF6F85eMT+MYVk4js+igirVG4S1IJhx0vrivh\ne69ujQW716BnulfBLtIGCndJGuuLAnzvtW1sLK5i4pAs/JW1hNph1UaR7kjhLgl3+NgJ/mv5Dl5a\nf4AhfTL52Y0z+eyskawvrtL0RpFzpHCXhDkZDPHoe/v49V/20BByfPmy8dx9+QSyMiP/LLvbvGSR\n9qRwl07nnONP28v4jze24a+o5YqpQ7ln4QWMye6d6NJEUobCXTrVnrJqvv/aNlbuPsKEIVk8+aV5\n/M3EwYkuSyTlKNylU6zYVc79f97F+qIqememce81U7n14lzSvZ5ElyaSkhTu0uFe33SQrz6zHkdk\nA4VfLprF5VOGJroskZSmZpN0qI3FVfzL85s+3CHHObaVVieyJJFuQS136TB/3VnGl59aR1YPL2Hn\nYjvzaM66SMdTuEuHeNFXwjdf3MSkoX14/I65FFfWac66SCdSuEu7cs7x0IpCfrx8B5eMH8RDt86h\nT490hvTpoVAX6UQKd2k34bDjh29s4zfv7+czM0fw089fqGV5RRJE4S7t4mQwxNef28gbm0q549Kx\n/NvVF+DxaKEvkURpdbaMmT1mZmVmtuUMj19mZkfNbEP06972L1OS2crd5VzxPyt4Y1Mp31k4he9e\no2AXSbR4Wu6PA78CnjjLOSudc9e0S0XSpby19RD/8KQPB6R7jTm5A7U0r0gSaLXl7pxbAVR2Qi3S\nxXyw9whfe3ZDbA57OOzIL6xIaE0iEtFeFzFdYmabzGy5mU1rp+eUJOWc45GVhdz66BoG9k4nM82D\n19AcdpEk0h4DquuAHOdcjZktBF4BJrZ0opktAZYA5OTktMNLS2erqw/xzRc38erGg3x62lB++vmZ\n7DpcoznsIknGnHOtn2Q2BnjdOTc9jnP3A3nOuSNnOy8vL88VFBTEV6UkhaKKWpY8WcDOw9X886cm\n8+WPjdfAqUgnMzOfcy6vtfPOu+VuZsOAw845Z2bziHT1qOM1xazYVc5Xl63HOcdjt83l8slDEl2S\niJxFq+FuZsuAy4BsMysB/h1IB3DOPQjcAHzZzIJAHbDIxfPngHQJzjkeeHcvP3lrJ5OH9uGhW+eQ\nO0ibaogku1bD3Tm3uJXHf0VkqqSkmJqTQf71hY28ufkQn5k5gv/63Ax6Zei6N5GuQO9UadFrGw/w\nvVe3UXm8nnsWXsCdfzNW89dFuhCFu5zmiVX7uff3WwHI8HqYnTtAwS7SxWizDmni9U0H+f6r22L3\nQ+GwLkwS6YIU7hLzyMpCvvLMeiYNzdKFSSJdnLplhHDY8Z9vbueR9/Zx1fRh/PymWWw9eEwXJol0\nYQr3bu5kMMQ3ntvI65tKue2SMXz3mql4Pcac3AEKdZEuTOHejR2ta+CuJwvIL6zk21dNYclHx2ng\nVCRFKNy7qdKjddz22FoKj9Rw/6JZXDtrZKJLEpF2pHDvhnYequa236yh+kSQx2+fx6UTshNdkoi0\nM4V7N+LzB3i+oJhXNx4kKzON5+66mKkj+ia6LBHpAAr3bsLnD7B4aT71oTAG/OzGmQp2kRSmee7d\nxMrd5dSHwgB4DPaWH09wRSLSkRTu3URheQ0QCXZdmCSS+tQt0w1sKqni9U2lXDF1KLNG99eFSSLd\ngMI9xTWEwnzzxc1kZ2XyPzfOpG+P9ESXJCKdQOGe4h5eWcj20mM8dOscBbtIN6I+9xS278hx7v/T\nbq6cNoxPTxuW6HJEpBMp3FOUc45vv7SJjDQP3792WqLLEZFOpnBPUc8VFJNfWMl3Fl7A0L49El2O\niHQyhXsKKjt2gvve2M78sQO5KW90ossRkQRQuKcYnz/AFx5dTW1DiB9dPwOPR6s8inRHmi2TQnz+\nAIuWrqIh5EjzGIHahkSXJCIJopZ7CvnLjsM0hBwQGVDV3qci3ZfCPUU451hdWAloiQERUbdMyniu\noJgCf4DbLsllcJ8eWmJApJtTuKcAf8Vxvv/aNi4eN4h7r5mmQVQRUbdMVxcKO77+3Ea8HuOnN85U\nsIsIoJZ7l/fgu3vx+QP84qZZjOzfM9HliEiSUMu9C9ty4Ci/+NMurr5wONfOGpHockQkiSjcu6gT\nDSH+3+82MKBXBvd9djpm6o4RkQ+pW6aL+slbO9ldVsNv75hH/14ZiS5HRJKMWu5d0OPv7+PR9/Zx\n1fRhfGzS4ESXIyJJqNVwN7PHzKzMzLac4XEzs1+a2R4z22Rms9u/TDll5e5yvvfaNgDe2VGGzx9I\ncEUikoziabk/Dlx5lsevAiZGv5YAD5x/WXIm//uX3bHbDaGwlhgQkRa1Gu7OuRVA5VlOuRZ4wkXk\nA/3NbHh7FSgfKq6sZZ2/Cq8ZXi0xICJn0R4DqiOB4kb3S6LHStvhuaWRn7y1kzSv8ctFF7G7rEZL\nDIjIGXXqbBkzW0Kk64acnJzOfOkub0NxFa9uPMhXPz6BT00bxqe0c56InEV7zJY5ADTe7mdU9Nhp\nnHNLnXN5zrm8wYM1yyNezjnue2Mb2VmZ3PWx8YkuR0S6gPYI91eBL0ZnzSwAjjrn1CXTjt7aepi1\n+wN8/YpJZGXq0gQRaV2rSWFmy4DLgGwzKwH+HUgHcM49CLwJLAT2ALXA7R1VbHdUHwzz4+XbmTgk\nixvzRiW6HBHpIloNd+fc4lYed8Dd7VaRNPH0aj/7K2r5ze1zSfPqmjMRiY/SIokdrWvg/j/v5iMT\nsrlMV6KKSBso3JPYr9/Zw9G6Br6z8AItDCYibaJwT1LLN5fy6Mp9XD5pMFNH9E10OSLSxSjck9Ca\nfRXc/cw6Qs7x/t4KrR8jIm2mcE8y9cEw97y8hbCL3A9q/RgROQeaNJ1E6oNh7n5mHbvLakjzGM45\nrR8jIudE4Z4kTgZD/ONT6/jzjjJ+cO00po3oR35hhdaPEZFzonBPAicaQvzDUz7+urOc//jsdL6w\nIBdAoS4i50zhnmAnGkIsedLHil3l/Oj6GSyepwXVROT8KdwTqK4+xN8/UcD7e4/w35+7kBvnjm79\nm0RE4qBwT5Da+iB3/raAVYUV/OSGmdwwR+vGiEj7UbgnwPGTQe54fC1r91fysxtnct1FCnYRaV8K\n907k8wdYsauct7ceYufhan5+0yyunTUy0WWJSApSuHcSnz/ALQ/ncyIYBuDrV0xSsItIh9EVqp3k\n3V1lsWD3GHg9WghMRDqOwr0THK1t4M1Nkc2pPAYZuupURDqYumU6WFVtPV94dDX+ylq+ddUUQmGn\nq05FpMMp3DtQ5fF6vvDIavaU1fDQrXP4+JShiS5JRLoJhXsHqag5yS2PrKbwyHGWfnEOl00ekuiS\nRKQbUbh3gPLqk9zySD7+iloe+7u5fGRidqJLEpFuRuHezsqOnWDxw/kcrDrBb26byyUTFOwi0vkU\n7u3o0NET3PxwPoeOneA3t8/VjBgRSRiFezt5e+sh/vXFTZyoD/HknfOZO2ZgoksSkW5M4d4OVuwq\n564nfThKP1jLAAAKbUlEQVQic9g9pguURCSxdBFTO/jR8u1EtzwlpD1PRSQJqOV+nl7deJDtpdXa\n81REkorC/TwcrKrj317ezEU5/fn2VVNYuz+gq09FJCko3M9ROOz45+c3Egw7fn7jLMZk92beWLXY\nRSQ5qM/9HD32/j4+2FvBd6+Zypjs3okuR0SkCYX7Odh5qJr/fmsnn7xgCIu076mIJCGFexudDIb4\np2fX0yczjR9dfyGmaY8ikoTU595GP3t7FzsOVfPIF/MY3Ccz0eWIiLQorpa7mV1pZjvNbI+ZfauF\nxy8zs6NmtiH6dW/7l5p4T6zaz0MrCrnigiF8cqqW7xWR5NVqy93MvMCvgSuAEmCtmb3qnNvW7NSV\nzrlrOqDGpLBsTRH3/n4rACt3H8HnD2jKo4gkrXha7vOAPc65QudcPfAscG3HlpVc/rqzjO++siV2\nv0FXoYpIkosn3EcCxY3ul0SPNXeJmW0ys+VmNq2lJzKzJWZWYGYF5eXl51Bu53t5fQl3/raAUQN6\nkpnmwWvoKlQRSXrtNaC6DshxztWY2ULgFWBi85Occ0uBpQB5eXmu+ePJ5uEVhdz35nYuHjeIpV+c\nw67DNeQXVugqVBFJevGE+wGg8WTuUdFjMc65Y41uv2lm/2dm2c65I+1TZucKhx0//sMOlq4oZOGM\nYfz8pllkpnmZkztAoS4iXUI83TJrgYlmNtbMMoBFwKuNTzCzYRad8G1m86LP2yU7pRtCYf75+Y0s\nXVHIFy/O5X8XzyYzzZvoskRE2qTVlrtzLmhmXwHeArzAY865rWb2D9HHHwRuAL5sZkGgDljknEv6\nbpfmauuD/OPT6/jrznK+ccUkvvLxCbpISUS6JEtUBufl5bmCgoKEvHZLKo/Xc8fja9lUUsV9181g\n8bycRJckInIaM/M55/JaO6/bX6Hq8wd4e+shXtt0kCM19TzwhTl8etqwRJclInJeunW4+/wBbn44\nn5PBMAA/vHaagl1EUkK3Xjhs5a7yWLB7DI6dCCa4IhGR9tGtw319cQCIBHuGLkwSkRTSbbtlni8o\n5t1dR/h83ijGDOqtC5NEJKV0y3Dffbiae3+/lQXjBvLj6y/E69F0RxFJLd2uW6auPsTdz6yjV4aX\n+xddpGAXkZTU7Vru33t1K7sO1/DEHfMY2rdHossREekQ3Sbcff4Aj723jzc2l/KPl43no5MGJ7ok\nEZEO0y3CvfF8djO4bLKCXURSW7foc1+190hsPrsBa/cHEluQiEgH6xbhXtcQAiLBrvnsItIdpHy3\nTFVtPcvWFDN5WB8+c+FwLh6frfnsIpLyUj7c//utnVTV1vPUl+YzdUTfRJcjItIpUrpbZkNxFcvW\nFHHbJWMV7CLSraRsuK/dX8ldTxbQv2c6/++K07ZzFRFJaSkZ7j5/gMVL8zl87CQ1J4PsOlyT6JJE\nRDpVSob7+3vKCYYjO0yFw478wi65nauIyDlLyXCvrY9MffQYpGvqo4h0Qyk3WyYYCrN8yyEmDO7N\ndbNHsmCcpj6KSPeTcuH+xuZS/BW1PHSr9kIVke4rpbplCvZX8sPXtzF6QE+uuGBoossREUmYlAl3\nnz/A4ofzOVJTz6FjJ1hfXJXokkREEiZlwv293eU0hDRDRkQEUijcD1TVAZohIyICKTKgunxzKS+u\nO8CMkX25cvpwbXYtIt1el2+5+/wB7n5mHaGwY9fhGgW7iAgpEO5vbT1E9GJUgqGw+tpFREiBcK/T\n1agiIqfp8n3uG0uqGD+4N9fPHqUuGRGRqC4d7ss3l7Kp5Ci3Lsjl7ssnJLocEZGkEVe3jJldaWY7\nzWyPmX2rhcfNzH4ZfXyTmc1u/1Kb8vkDfHXZegCeKyjG59em1yIip7Qa7mbmBX4NXAVMBRab2dRm\np10FTIx+LQEeaOc6T5NfeCS2rK8GUkVEmoqnW2YesMc5VwhgZs8C1wLbGp1zLfCEc84B+WbW38yG\nO+dK27tgnz9AfmEFO0qrATA0kCoi0lw84T4SKG50vwSYH8c5I4F2DffIDkurqI8uMwDg9Rj3XjNN\nA6kiIo106lRIM1tiZgVmVlBeXt7m788vrIh1xZzinCNQW99eJYqIpIR4wv0AMLrR/VHRY209B+fc\nUudcnnMub/DgwW2tlQXjBpGR5okVrbntIiIti6dbZi0w0czGEgnsRcDNzc55FfhKtD9+PnC0I/rb\n5+QO4Ok7F5BfWMGAXhkEaus1t11EpAWthrtzLmhmXwHeArzAY865rWb2D9HHHwTeBBYCe4Ba4PaO\nKnhO7gCFuYhIK+K6iMk59yaRAG987MFGtx1wd/uWJiIi56rLry0jIiKnU7iLiKQghbuISApSuIuI\npCCFu4hICrLIRJcEvLBZOeA/x2/PBo60YzmdSbUnRletvavWDaq9o+Q651q9CjRh4X4+zKzAOZeX\n6DrOhWpPjK5ae1etG1R7oqlbRkQkBSncRURSUFcN96WJLuA8qPbE6Kq1d9W6QbUnVJfscxcRkbPr\nqi13ERE5i6QO92TcmDtecdR+S7TmzWb2gZnNTESdzbVWd6Pz5ppZ0Mxu6Mz6ziae2s3sMjPbYGZb\nzezdzq7xTOL499LPzF4zs43R2jts5dW2MLPHzKzMzLac4fFkfo+2VntSvkfj5pxLyi8iywvvBcYB\nGcBGYGqzcxYCy4lspboAWJ3outtQ+yXAgOjtq5Kh9njqbnTeX4isFHpDoutuw++8P5G9f3Oi94ck\nuu421P4d4L+itwcDlUBGEtT+UWA2sOUMjyflezTO2pPuPdqWr2Ruucc25nbO1QOnNuZuLLYxt3Mu\nH+hvZsM7u9AWtFq7c+4D51wgejefyO5ViRbP7xzgq8CLQFlnFteKeGq/GXjJOVcE4JxLlvrjqd0B\nfczMgCwi4R7s3DJP55xbEa3lTJL1Pdpq7Un6Ho1bMof7mTbdbus5idDWur5EpHWTaK3WbWYjgeuA\nBzqxrnjE8zufBAwws7+amc/Mvthp1Z1dPLX/CrgAOAhsBv7JORfunPLOS7K+R9sqWd6jcYtrsw7p\nOGZ2OZF/OB9JdC1x+gXwTedcONKI7FLSgDnAJ4CewCozy3fO7UpsWXH5NLAB+DgwHvijma10zh1L\nbFmprwu+R4HkDvd225g7AeKqy8wuBB4BrnLOVXRSbWcTT915wLPRYM8GFppZ0Dn3SueUeEbx1F4C\nVDjnjgPHzWwFMBNIdLjHU/vtwI9dpAN4j5ntA6YAazqnxHOWrO/RuCThezR+ie70P8tgRxpQCIzl\nw0Gmac3OuZqmgzVrEl13G2rPIbLn7CWJrrctdTc7/3GSZ0A1nt/5BcCfo+f2ArYA07tI7Q8A34ve\nHkokILMTXXu0njGceVAyKd+jcdaedO/RtnwlbcvdJdnG3G0RZ+33AoOA/4u2goMuwQsVxVl3Uoqn\ndufcdjP7A7AJCAOPOOdanAbXmeL8vf8QeNzMNhMJym865xK+aqGZLQMuA7LNrAT4dyAdkvs9CnHV\nnnTv0bbQFaoiIikomWfLiIjIOVK4i4ikIIW7iEgKUriLiKQghbuISApSuIuIpCCFu4hIClK4i4ik\noP8fpQuya24IEV4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8e8239f890>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import clear_output",
    "\n",
    "\n",
    "s.run(tf.global_variables_initializer())",
    "\n",
    "\n",
    "guesses = [s.run(y_guess)]",
    "\n",
    "\n",
    "for _ in range(100):",
    "\n",
    "    s.run(optimizer)",
    "\n",
    "    guesses.append(s.run(y_guess))",
    "\n",
    "\n",
    "    clear_output(True)",
    "\n",
    "    plt.plot(*zip(*guesses), marker='.')",
    "\n",
    "    plt.scatter(*s.run(y_true), c='red')",
    "\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic regression example\n",
    "__[ 4 points max]__\n",
    "\n",
    "Implement the regular logistic regression training algorithm\n",
    "\n",
    "Tips:\n",
    "* Weights fit in as a shared variable\n",
    "* X and y are potential inputs\n",
    "* Compile 2 functions:\n",
    " * train_function(X,y) - returns error and computes weights' new values __(through updates)__\n",
    " * predict_fun(X) - just computes probabilities (\"y\") given data\n",
    " \n",
    " \n",
    "We shall train on a two-class MNIST dataset\n",
    "* please note that target y are {0,1} and not {-1,1} as in some formulae"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits",
    "\n",
    "mnist = load_digits(2)",
    "\n",
    "\n",
    "X, y = mnist.data, mnist.target",
    "\n",
    "\n",
    "\n",
    "print \"y [shape - %s]:\" % (str(y.shape)), y[:10]",
    "\n",
    "\n",
    "print \"X [shape - %s]:\" % (str(X.shape))",
    "\n",
    "print X[:3]",
    "\n",
    "print y[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# inputs and shareds",
    "\n",
    "weights = <student.code_variable() >",
    "\n",
    "input_X = <student.code_placeholder() >",
    "\n",
    "input_y = <student.code_placeholder() >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted_y = <predicted probabilities for input_X >",
    "\n",
    "loss = <logistic loss(scalar, mean over sample) >",
    "\n",
    "\n",
    "optimizer = <optimizer that minimizes loss >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_function = <compile function that takes X and y, returns log loss and updates weights >",
    "\n",
    "predict_function = <compile function that takes X and computes probabilities of y >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cross_validation import train_test_split",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score",
    "\n",
    "\n",
    "for i in range(5):",
    "\n",
    "    <run optimizer operation >",
    "\n",
    "    loss_i = <compute loss at iteration i >",
    "\n",
    "\n",
    "    print(\"loss at iter %i:%.4f\" % (i, loss_i))",
    "\n",
    "\n",
    "    print(\"train auc:\", roc_auc_score(y_train, predict_function(X_train)))",
    "\n",
    "    print(\"test auc:\", roc_auc_score(y_test, predict_function(X_test)))",
    "\n",
    "\n",
    "\n",
    "print(\"resulting weights:\")",
    "\n",
    "plt.imshow(shared_weights.get_value().reshape(8, -1))",
    "\n",
    "plt.colorbar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# my1stNN\n",
    "__[basic part 4 points max]__\n",
    "Your ultimate task for this week is to build your first neural network [almost] from scratch and pure tensorflow.\n",
    "\n",
    "This time you will same digit recognition problem, but at a larger scale\n",
    "* images are now 28x28\n",
    "* 10 different digits\n",
    "* 50k samples\n",
    "\n",
    "Note that you are not required to build 152-layer monsters here. A 2-layer (one hidden, one output) NN should already have ive you an edge over logistic regression.\n",
    "\n",
    "__[bonus score]__\n",
    "If you've already beaten logistic regression with a two-layer net, but enthusiasm still ain't gone, you can try improving the test accuracy even further! The milestones would be 95%/97.5%/98.5% accuraсy on test set.\n",
    "\n",
    "__SPOILER!__\n",
    "At the end of the notebook you will find a few tips and frequently made mistakes. If you feel enough might to shoot yourself in the foot without external assistance, we encourage you to do so, but if you encounter any unsurpassable issues, please do look there before mailing us.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mnist import load_dataset",
    "\n",
    "\n",
    "# [down]loading the original MNIST dataset.",
    "\n",
    "# Please note that you should only train your NN on _train sample,",
    "\n",
    "# _val can be used to evaluate out-of-sample error, compare models or perform early-stopping",
    "\n",
    "# _test should be hidden under a rock untill final evaluation... But we both know it is near impossible to catch you evaluating on it.",
    "\n",
    "X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()",
    "\n",
    "\n",
    "print X_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(X_train[0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<here you could just as well create computation graph >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<this may or may not be a good place to evaluating loss and optimizer >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<this may be a perfect cell to write a training & evaluation loop in >"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "<predict & evaluate on test here, right? No cheating pls. >"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Report\n",
    "\n",
    "I did such and such, that did that cool thing and my stupid NN bloated out that stuff. Finally, i did that thingy and felt like Le'Cun. That cool article and that kind of weed helped me so much (if any)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "# SPOILERS!\n",
    "\n",
    "Recommended pipeline\n",
    "\n",
    "* Adapt logistic regression from previous assignment to classify some number against others (e.g. zero vs nonzero)\n",
    "* Generalize it to multiclass logistic regression.\n",
    "  - Either try to remember lecture 0 or google it.\n",
    "  - Instead of weight vector you'll have to use matrix (feature_id x class_id)\n",
    "  - softmax (exp over sum of exps) can implemented manually or as T.nnet.softmax (stable)\n",
    "  - probably better to use STOCHASTIC gradient descent (minibatch)\n",
    "    - in which case sample should probably be shuffled (or use random subsamples on each iteration)\n",
    "* Add a hidden layer. Now your logistic regression uses hidden neurons instead of inputs.\n",
    "  - Hidden layer uses the same math as output layer (ex-logistic regression), but uses some nonlinearity (sigmoid) instead of softmax\n",
    "  - You need to train both layers, not just output layer :)\n",
    "  - Do not initialize layers with zeros (due to symmetry effects). A gaussian noize with small sigma will do.\n",
    "  - 50 hidden neurons and a sigmoid nonlinearity will do for a start. Many ways to improve. \n",
    "  - In ideal casae this totals to 2 .dot's, 1 softmax and 1 sigmoid\n",
    "  - __make sure this neural network works better than logistic regression__\n",
    "  \n",
    "* Now's the time to try improving the network. Consider layers (size, neuron count),  nonlinearities, optimization methods, initialization - whatever you want, but please avoid convolutions for now.\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
